Using MemoryMappedBuffer to handle large file in Java

When handling large files, it will largely affect the process speed while using traditional FileInputStream, FileOutputStream or RandomAccessFile since they trigger lots of read and write operations. In Java NIO, a new way of handling large file is introduced which is to use MmeoryMappedBuffer to create memory mapped file.

Memory-mapped I/O uses the filesystem to establish a virtual memory mapping from user space directly to the applicable filesystem pages. With a memory-mapped file, you can pretend that the entire file is in memory and that you can access it by simply treating it as a very large array. And since it will not trigger lots read and write operation, it can greatly increase the processing speed.

Below is a test which will demonstrate the processing speed for the traditional IO and memory mapped IO.

import java.io.BufferedInputStream;  
import java.io.FileInputStream;  
import java.io.FileNotFoundException;  
import java.io.IOException;  
import java.io.RandomAccessFile;  
import java.nio.MappedByteBuffer;  
import java.nio.channels.FileChannel;  

/**
 * The test.txt is a large file which has 8.1 MB
 */
public class Test {  
    public static void main(String[] args) {
    	String filename = "test.txt";
    	
    	// Traditional IO using FileInputStream
        try {  
            FileInputStream fis=new FileInputStream(filename);  
            int sum=0;  
            int n;  
            long t1=System.currentTimeMillis();  
            try {  
                while((n=fis.read())>=0){  
                    sum+=n;  
                }  
            } catch (IOException e) {  
                // TODO Auto-generated catch block  
                e.printStackTrace();  
            }  
            long t=System.currentTimeMillis()-t1;  
            System.out.println("FileInputStream sum:"+sum+"  time:"+t);  
        } catch (FileNotFoundException e) {  
            // TODO Auto-generated catch block  
            e.printStackTrace();  
        }  
        
        // Traditional IO using BufferedInputStream
        try {  
            FileInputStream fis=new FileInputStream(filename);  
            BufferedInputStream bis=new BufferedInputStream(fis);  
            int sum=0;  
            int n;  
            long t1=System.currentTimeMillis();  
            try {  
                while((n=bis.read())>=0){  
                    sum+=n;  
                }  
            } catch (IOException e) {  
                // TODO Auto-generated catch block  
                e.printStackTrace();  
            }  
            long t=System.currentTimeMillis()-t1;  
            System.out.println("BufferedInputStream sum:"+sum+"  time:"+t);  
        } catch (FileNotFoundException e) {  
            // TODO Auto-generated catch block  
            e.printStackTrace();  
        }  

        // NIO way of using memory mapped buffer
        MappedByteBuffer buffer=null;  
        int length = 1253244;
        try {  
            buffer=new RandomAccessFile(filename,"rw").getChannel().map(FileChannel.MapMode.READ_WRITE, 0, length);  
            int sum=0;  
            int n;  
            long t1=System.currentTimeMillis();  
            for(int i=0;i<length;i++){  
                n=0x000000ff&buffer.get(i);  
                sum+=n;  
            }  
            long t=System.currentTimeMillis()-t1;  
            System.out.println("MmoeryMappedBuffer sum:"+sum+"  time:"+t);  
        } catch (FileNotFoundException e) {  
            // TODO Auto-generated catch block  
            e.printStackTrace();  
        } catch (IOException e) {  
            // TODO Auto-generated catch block  
            e.printStackTrace();  
        }  
    }  
}

Here we are trying to read a file with 8.1 MB. Below is the time taken for each read.

FileInputStream sum:618000000  time:21478
BufferedInputStream sum:618000000  time:229
MmoeryMappedBuffer sum:91118309  time:44

From the output, we can see that it takes much less time to read the file using MemoryMappedBuffer.

But be cautious that all or part of a mapped byte buffer may become inaccessible at any time, for example if the mapped file is truncated. An attempt to access an inaccessible region of a mapped byte buffer will not change the buffer's content and will cause an unspecified exception to be thrown either at the time of the access or at some later time. It is therefore strongly recommended that appropriate precautions be taken to avoid the manipulation of a mapped file by this program, or by a concurrently running program, except to read or write the file's content.

Reference : Java使用内存映射实现大文件的上传