MySQL BinLog

Reading and Writing to a Random Access File:

public class RandomAccessFile

A random access file behaves like a large array of bytes stored in the file system.  There is a "cursor", or index into the implied array, called the file pointer.  

Input operations read bytes starting at the pointer and advance the file pointer past the bytes read.  Same for output operations.  Output operations that write past the current end of the implied array cause the array to be extended.  

The file pointer can be read by the "getFilePointer" method and set by the "seek" method.

If end-of-file is reached before the desired number of bytes has been read, an EOFException (a kind of IOException) is thrown.  If any byte cannot be read (ex: stream has been closed), an IOException is thrown.

Common Methods:

public int read(bytes[] b, int offset, int len) throws IOException

// Reads up to "len" bytes of data from this file into an array of bytes.  This methods blocks until at least one byte of input is available

// Returns the total number of bytes read into the buffer, or -1 if there is no more data because the end of file has been reached

public long getFilePointer() throws IOException

// returns the current offset in this file

USE MYSQL binlog to process the incremental data， instead of DBDump

A database dump contains a record of the table structure and the data from a database and is usually in the form of a list of SQL statements.  

MySQL has "the general query log", "the slow query log", "the error log", and the "binary log".  

The general query log: a general record of what mysqld (MySQL Server) is doing.  The server writes information to this log when clients connect or disconnect, and it logs each SQL statement received from clients.  The log can be useful when you suspect an error in a client and want to know exactly what the client sent to mysqld.

The slow query log: consists of SQL statements that took more than "long_query_time" seconds to execute.  The minimum value of "long_query_time" is 1, and default is 10.

The error log: contains information indicating when mysqld was started and stopped, also any critical errors that occur while the server is running.  If mysqld notices a table that needs to be automatically checked or repaired, it writes a message to the error log

The binary log: contains "events" that describe database changes such as table creation operations or changes to table data (如create,insert,drop,update等).  Also contains info about how long each statement took that updated data.  The binary log has two purposes:

1. For replication.  The binary log on a master replication server provides a record of the data changes to be sent to slave servers.  The master server sends the events contained in its binary log to its slaves, which execute those events to make the same data changes that were made on the master

2. Data recovery.  

binlog_format参数在Mysql5.1之后可以用STATEMENT,ROW，MIXED三种模式。STATEMENT是Mysql一直以来支持的格式，记录的是日志的逻辑语句。ROW格式下就可以记录行的更改情况，对于数据复制和恢复的可靠性更好，但同时也会增加二进制文件的大小。MIXED格式是综合了前两者，具体要看存储引擎的支持情况。InnoDB和MyISAM是STATEMENT,ROW两者都支持的。

Mysql官方有提供了mysqlbinlog 工具来查看日志。 把binlog分成几个模块来看的话，开始的部分都是由0xfe

0x62 0x69 0x6e 4个字节组成的魔数，接下来的部分就是Common-Header，再后面就是body部分

1. 先读固定的魔术组：

public InputStream parseInit(String path) throws Exception { 

    InputStream inputStream = new RandomAccessFileInputStream(new File(path)); 

    try { 

        final byte[] binlogMagic = new byte[] {(byte)0xfe, (byte)0x62, (byte)0x69, (byte)0x6e};

        final byte[] inputMagic = new byte[binlogMagic.length]；

        inputStream.read(inputMagic, 0, binlogMagic.length); 

        if (!Arrays.equals(inputMagic, binlogMagic)) {

            throw new Exception(); 

        } 

        return inputStream; 

    } catch (Exception e) {

        inputStream.close(); 

        throw e; 

    } 

}

2. Read in the common header:

long timeStamp = inputStream.readLong(4); // 操作前的时间戳

int type = inputStream.readInt(1); // 事件类型，标记了操作类型，事务提交或者查询等。

long serverId = inputStream.readLong(4); // 事件创建的server id。

int eventLength = inputStream.readInt(4); // 该条记录的size

long nextPosition = inputStream.readLong(4);