dev.advo.fs.fs.IndexedFileSystem Class Reference

A file system based on top of the operating system's file system. More...

Inheritance diagram for dev.advo.fs.fs.IndexedFileSystem:

Collaboration diagram for dev.advo.fs.fs.IndexedFileSystem:

Public Member Functions
void	close () throws IOException
ByteBuffer	getCrcTable () throws IOException
	Gets the CRC table.
ByteBuffer	getFile (FileDescriptor fd) throws IOException
	Gets a file.
ByteBuffer	getFile (int type, int file) throws IOException
	Gets a file.
ByteBuffer	getJireCRCTable () throws IOException
	IndexedFileSystem (File base, boolean readOnly) throws Exception
	Creates the file system with the specified base directory.
boolean	isReadOnly ()
	Checks if this IndexedFileSystem is read only.

Static Public Member Functions
static byte[]	gzip (byte[] bytes) throws IOException
	Compresses a GZIP file.

Static Public Attributes
static byte[]	bytes

Private Member Functions
void	detectLayout (File base) throws Exception
	Automatically detect the layout of the specified directory.
int	getFileCount (int type) throws IOException
	Gets the number of files with the specified type.
Index	getIndex (FileDescriptor fd) throws IOException
	Gets the index of a file.
byte[]	initialResponse () throws IOException

Private Attributes
ByteBuffer	crcTable
	The cached CRC table.
RandomAccessFile	data
	The data file.
RandomAccessFile[]	indices = new RandomAccessFile[256]
	The index files.
final boolean	readOnly
	Read only flag.

Detailed Description

A file system based on top of the operating system's file system.

It consists of a data file and index files. Index files point to blocks in the data file, which contains the actual data.

Author

Graham

Definition at line 20 of file IndexedFileSystem.java.

Constructor & Destructor Documentation

IndexedFileSystem()

dev.advo.fs.fs.IndexedFileSystem.IndexedFileSystem	(	File	base,
		boolean	readOnly ) throws Exception

Creates the file system with the specified base directory.

Parameters

base	The base directory.
readOnly	A flag indicating if the file system will be read only.

Exceptions

Exception

if the file system is invalid.

Definition at line 48 of file IndexedFileSystem.java.

                                                                           {
        this.readOnly = readOnly;
        detectLayout(base);
    }

References detectLayout(), and readOnly.

Here is the call graph for this function:

Member Function Documentation

close()

void dev.advo.fs.fs.IndexedFileSystem.close

(

)

throws IOException

Definition at line 443 of file IndexedFileSystem.java.

                                           {
        if (data != null) {
            synchronized (data) {
                data.close();
            }
        }
        
        for (RandomAccessFile index : indices) {
            if (index != null) {
                synchronized (index) {
                    index.close();
                }
            }
        }
    }

References data, and indices.

detectLayout()

void dev.advo.fs.fs.IndexedFileSystem.detectLayout ( File base ) throws Exception

private

Automatically detect the layout of the specified directory.

Parameters

base	The base directory.

Exceptions

Exception

if the file system is invalid.

Definition at line 66 of file IndexedFileSystem.java.

                                                          {
        int indexCount = 0;
        for (int index = 0; index < indices.length; index++) {
            File f = new File(base.getAbsolutePath() + "/main_file_cache.idx" + index);
            if (f.exists() && !f.isDirectory()) {
                indexCount++;
                indices[index] = new RandomAccessFile(f, readOnly ? "r" : "rw");
            }
        }
        if (indexCount <= 0) {
            throw new Exception("No index file(s) present");
        }
        
        File oldEngineData = new File(base.getAbsolutePath() + "/main_file_cache.dat");
        File newEngineData = new File(base.getAbsolutePath() + "/main_file_cache.dat2");
        if (oldEngineData.exists() && !oldEngineData.isDirectory()) {
            data = new RandomAccessFile(oldEngineData, readOnly ? "r" : "rw");
        } else if (newEngineData.exists() && !oldEngineData.isDirectory()) {
            data = new RandomAccessFile(newEngineData, readOnly ? "r" : "rw");
        } else {
            throw new Exception("No data file present");
        }
        if(bytes == null) {
            bytes = initialResponse();
        }
    }

References bytes, data, indices, initialResponse(), and readOnly.

Referenced by IndexedFileSystem().

Here is the call graph for this function:

Here is the caller graph for this function:

getCrcTable()

ByteBuffer dev.advo.fs.fs.IndexedFileSystem.getCrcTable

(

)

throws IOException

Gets the CRC table.

Returns

The CRC table.

Exceptions

IOException

if an I/O erorr occurs.

Definition at line 144 of file IndexedFileSystem.java.

                                                       {
        if (readOnly) {
            synchronized (this) {
                if (crcTable != null) {
                    return crcTable.slice();
                }
            }
            
            // the number of archives
            int archives = getFileCount(0);
            
            // the hash
            int hash = 1234;
            
            // the CRCs
            int[] crcs = new int[archives];
            
            // calculate the CRCs
            CRC32 crc32 = new CRC32();                      
            for (int i = 1; i < crcs.length; i++) {
                crc32.reset();
                
                ByteBuffer bb = getFile(0, i);
                byte[] bytes = new byte[bb.remaining()];
                bb.get(bytes, 0, bytes.length);
                crc32.update(bytes, 0, bytes.length);
                
                crcs[i] = (int) crc32.getValue();
            }
            
            // hash the CRCs and place them in the buffer
            ByteBuffer buf = ByteBuffer.allocate(crcs.length * 4 + 4);
            for (int i = 0; i < crcs.length; i++) {
                hash = (hash << 1) + crcs[i];
                buf.putInt(crcs[i]);
            }
            
            // place the hash into the buffer
            buf.putInt(hash);
            buf.flip();
            
            synchronized (this) {
                crcTable = buf;
                return crcTable.slice();
            }
        } else {
            throw new IOException("cannot get CRC table from a writable file system");
        }
    }

References bytes, crcTable, getFile(), getFileCount(), and readOnly.

Here is the call graph for this function:

getFile() [1/2]

ByteBuffer dev.advo.fs.fs.IndexedFileSystem.getFile

(

FileDescriptor

fd

)

throws IOException

Gets a file.

Parameters

fd	The FileDescriptor which points to the file.

Returns

A ByteBuffer which contains the contents of the file.

Exceptions

IOException

if an I/O error occurs.

Definition at line 371 of file IndexedFileSystem.java.

                                                                    {
        Index index = getIndex(fd);
        ByteBuffer buffer = ByteBuffer.allocate(index.getSize());
        
        // calculate some initial values
        long ptr = (long) index.getBlock() * (long) FileSystemConstants.BLOCK_SIZE;
        int read = 0;
        int size = index.getSize();
        int blocks = size / FileSystemConstants.CHUNK_SIZE;
        if (size % FileSystemConstants.CHUNK_SIZE != 0) {
            blocks++;
        }
        
        for (int i = 0; i < blocks; i++) {
            
            // read header
            byte[] header = new byte[FileSystemConstants.HEADER_SIZE];
            synchronized (data) {
                data.seek(ptr);
                data.readFully(header);
            }
            
            // increment pointers
            ptr += FileSystemConstants.HEADER_SIZE;
            
            // parse header
            int nextFile = ((header[0] & 0xFF) << 8) | (header[1] & 0xFF);
            int curChunk = ((header[2] & 0xFF) << 8) | (header[3] & 0xFF);
            int nextBlock = ((header[4] & 0xFF) << 16) | ((header[5] & 0xFF) << 8) | (header[6] & 0xFF);
            int nextType = header[7] & 0xFF;
            
            // check expected chunk id is correct
            if (i != curChunk) {
                throw new IOException("Chunk id mismatch.");
            }
            
            // calculate how much we can read
            int chunkSize = size - read;
            if (chunkSize > FileSystemConstants.CHUNK_SIZE) {
                chunkSize = FileSystemConstants.CHUNK_SIZE;
            }
            
            // read the next chunk and put it in the buffer
            byte[] chunk = new byte[chunkSize];
            synchronized (data) {
                data.seek(ptr);
                data.readFully(chunk);
            }           
            buffer.put(chunk);
            
            // increment pointers
            read += chunkSize;
            ptr = (long) nextBlock * (long) FileSystemConstants.BLOCK_SIZE;
            
            // if we still have more data to read, check the validity of the
            // header
            if (size > read) {              
                if (nextType != (fd.getType() + 1)) {
                    throw new IOException("File type mismatch.");
                }
                
                if (nextFile != fd.getFile()) {
                    throw new IOException("File id mismatch.");
                }
            }
        }
        
        buffer.flip();
        return buffer;
    }

References dev.advo.fs.fs.FileSystemConstants.BLOCK_SIZE, dev.advo.fs.fs.FileSystemConstants.CHUNK_SIZE, data, getIndex(), and dev.advo.fs.fs.FileSystemConstants.HEADER_SIZE.

Here is the call graph for this function:

getFile() [2/2]

ByteBuffer dev.advo.fs.fs.IndexedFileSystem.getFile	(	int	type,
		int	file ) throws IOException

Gets a file.

Parameters

type	The file type.
file	The file id.

Returns

A ByteBuffer which contains the contents of the file.

Exceptions

IOException

if an I/O error occurs.

Definition at line 361 of file IndexedFileSystem.java.

                                                                     {
        return getFile(new FileDescriptor(type, file));
    }

References getFile().

Referenced by getCrcTable(), getFile(), getJireCRCTable(), and initialResponse().

Here is the call graph for this function:

Here is the caller graph for this function:

getFileCount()

int dev.advo.fs.fs.IndexedFileSystem.getFileCount ( int type ) throws IOException

private

Gets the number of files with the specified type.

Parameters

type

The type.

Returns

The number of files.

Exceptions

IOException

if an I/O error occurs.

Definition at line 128 of file IndexedFileSystem.java.

                                                          {
        if (type < 0 || type >= indices.length) {
            throw new IndexOutOfBoundsException();
        }
        
        RandomAccessFile indexFile = indices[type];
        synchronized (indexFile) {
            return (int) (indexFile.length() / FileSystemConstants.INDEX_SIZE);
        }
    }

References dev.advo.fs.fs.FileSystemConstants.INDEX_SIZE, and indices.

Referenced by getCrcTable(), getJireCRCTable(), and initialResponse().

Here is the caller graph for this function:

getIndex()

Index dev.advo.fs.fs.IndexedFileSystem.getIndex ( FileDescriptor fd ) throws IOException

private

Gets the index of a file.

Parameters

fd	The FileDescriptor which points to the file.

Returns

The Index.

Exceptions

IOException

if an I/O error occurs.

Definition at line 101 of file IndexedFileSystem.java.

                                                                 {
        int index = fd.getType();
        if (index < 0 || index >= indices.length) {
            throw new IndexOutOfBoundsException();
        }
        
        byte[] buffer = new byte[FileSystemConstants.INDEX_SIZE];
        RandomAccessFile indexFile = indices[index];
        synchronized (indexFile) {
            long ptr = (long) fd.getFile() * (long) FileSystemConstants.INDEX_SIZE;
            if (ptr >= 0 && indexFile.length() >= (ptr + FileSystemConstants.INDEX_SIZE)) {
                indexFile.seek(ptr);
                indexFile.readFully(buffer);
            } else {
                throw new FileNotFoundException();
            }
        }
                
        return Index.decode(buffer);
    }

References dev.advo.fs.fs.Index.decode(), dev.advo.fs.fs.FileSystemConstants.INDEX_SIZE, and indices.

Referenced by getFile().

Here is the call graph for this function:

Here is the caller graph for this function:

getJireCRCTable()

ByteBuffer dev.advo.fs.fs.IndexedFileSystem.getJireCRCTable

(

)

throws IOException

Definition at line 302 of file IndexedFileSystem.java.

                                                           {
        if (readOnly) {
            synchronized (this) {
                if (crcTable != null) {
                    return crcTable.slice();
                }
            }
            
            int size = 6;
            
            // the number of archives
            int archives = getFileCount(0);
            
            // the hash
            int hash = 1234;
            
            // the CRCs
            int[] crcs = new int[archives];
            
            // calculate the CRCs
            CRC32 crc32 = new CRC32();                      
            for (int i = 1; i < crcs.length; i++) {
                crc32.reset();
                
                ByteBuffer bb = getFile(0, i);
                byte[] bytes = new byte[bb.remaining()];
                bb.get(bytes, 0, bytes.length);
                crc32.update(bytes, 0, bytes.length);
                
                crcs[i] = (int) crc32.getValue();
            }
            
            // hash the CRCs and place them in the buffer
            ByteBuffer buf = ByteBuffer.allocate(crcs.length * 4 + 4);
            for (int i = 0; i < crcs.length; i++) {
                hash = (hash << 1) + crcs[i];
                buf.putInt(crcs[i]);
            }
            
            // place the hash into the buffer
            buf.putInt(hash);
            buf.flip();
            
            synchronized (this) {
                crcTable = buf;
                return crcTable.slice();
            }
        } else {
            throw new IOException("cannot get CRC table from a writable file system");
        }
    }

References bytes, crcTable, getFile(), getFileCount(), and readOnly.

Here is the call graph for this function:

gzip()

byte[] dev.advo.fs.fs.IndexedFileSystem.gzip ( byte[] bytes ) throws IOException

static

Compresses a GZIP file.

Parameters

bytes

The uncompressed bytes.

Returns

The compressed bytes.

Exceptions

IOException

if an I/O error occurs.

Definition at line 203 of file IndexedFileSystem.java.

                                                               {
        /* create the streams */
        InputStream is = new ByteArrayInputStream(bytes);
        try {
            ByteArrayOutputStream bout = new ByteArrayOutputStream();
            OutputStream os = new GZIPOutputStream(bout);
            try {
                /* copy data between the streams */
                byte[] buf = new byte[4096];
                int len = 0;
                while ((len = is.read(buf, 0, buf.length)) != -1) {
                    os.write(buf, 0, len);
                }
            } finally {
                os.close();
            }
 
            /* return the compressed bytes */
            return bout.toByteArray();
        } finally {
            is.close();
        }
    }

References bytes.

Referenced by initialResponse().

Here is the caller graph for this function:

initialResponse()

byte[] dev.advo.fs.fs.IndexedFileSystem.initialResponse ( ) throws IOException

private

Definition at line 239 of file IndexedFileSystem.java.

                                                        {
        // calculate the CRCs
        CRC32 crc32 = new CRC32();  
//      {
//        ChannelBuffer channelBuffer = ChannelBuffers.dynamicBuffer();
//        channelBuffer.writeByte(6);
//        for(int type = 0; type < 6; type++) {
//          int files = getFileCount(type);
//          channelBuffer.writeMedium(files);
//          
//          for(int file = 0; file < files; file++) {
//              crc32.reset();
//              ByteBuffer bb = getFile(type, file);
//              byte[] bytes = new byte[bb.remaining()];
//              bb.get(bytes, 0, bytes.length);
//              crc32.update(bytes, 0, bytes.length);
//          }
//
//          //write int for the crc value.
//          channelBuffer.writeInt((int) crc32.getValue());
//        }
//      }
//      {
        int bufLen = 0;
        for(int i = 0; i < 6; i++) {
            bufLen += getFileCount(i) * 4;
        }
        bufLen += (6*3) + 1;
 
        ByteBuffer buf = ByteBuffer.allocate(bufLen);
        buf.put((byte) 6);
 
        for(int i = 0; i < 6; i++) {
            //write the # of entries as a medium.
            int value = getFileCount(i);
            buf.put((byte) (value >> 16));
            buf.put((byte) (value >> 8));
            buf.put((byte) value);
            for(int f = 0; f < value; f++) {
                crc32.reset();
                ByteBuffer bb = getFile(i, f);
                byte[] bytes = new byte[bb.remaining()];
                bb.get(bytes, 0, bytes.length);
                crc32.update(bytes, 0, bytes.length);
 
                //write int for the crc value.
                buf.putInt((int) crc32.getValue());
            }
        }
 
        buf.rewind();
        byte[] uncompressed = new byte[buf.remaining()];
        //System.out.println("unc" + uncompressed.length);
        buf.get(uncompressed);
//      System.out.println(Arrays.toString(uncompressed));
//      System.exit(0);
        byte[] compressed = gzip(uncompressed);
        System.out.println("comp" + compressed.length);
//      }
 
        return compressed;
    }

References bytes, getFile(), getFileCount(), and gzip().

Referenced by detectLayout().

Here is the call graph for this function:

Here is the caller graph for this function:

isReadOnly()

boolean dev.advo.fs.fs.IndexedFileSystem.isReadOnly

(

)

Checks if this IndexedFileSystem is read only.

Returns

true if so, false if not.

Definition at line 57 of file IndexedFileSystem.java.

                                {
        return readOnly;
    }

References readOnly.

Member Data Documentation

bytes

byte [] dev.advo.fs.fs.IndexedFileSystem.bytes

static

Definition at line 93 of file IndexedFileSystem.java.

Referenced by detectLayout(), dev.advo.fs.net.service.ServiceResponseEncoder.encode(), getCrcTable(), getJireCRCTable(), gzip(), and initialResponse().

crcTable

ByteBuffer dev.advo.fs.fs.IndexedFileSystem.crcTable

private

The cached CRC table.

Definition at line 40 of file IndexedFileSystem.java.

Referenced by getCrcTable(), and getJireCRCTable().

data

RandomAccessFile dev.advo.fs.fs.IndexedFileSystem.data

private

The data file.

Definition at line 35 of file IndexedFileSystem.java.

Referenced by close(), detectLayout(), and getFile().

indices

RandomAccessFile [] dev.advo.fs.fs.IndexedFileSystem.indices = new RandomAccessFile[256]

private

The index files.

Definition at line 30 of file IndexedFileSystem.java.

Referenced by close(), detectLayout(), getFileCount(), and getIndex().

readOnly

final boolean dev.advo.fs.fs.IndexedFileSystem.readOnly

private

Read only flag.

Definition at line 25 of file IndexedFileSystem.java.

Referenced by detectLayout(), getCrcTable(), getJireCRCTable(), IndexedFileSystem(), and isReadOnly().

---

The documentation for this class was generated from the following file:

• java/dev/advo/fs/fs/IndexedFileSystem.java