secondNamenode chkpiont的理解

每次进行写操作，edits都要进行更新和同步数据（这点与oracle的redo日志很相似）下面的测试说明这一点：

写数据前edits的大小：

[grid@h1 current]$ ll

总计 16

-rw-rw-r-- 1 grid grid    4 08-24 12:24 edits

写入数据：

[grid@h1 current]$ hadoop fs -put VERSION in 

[grid@h1 current]$ ll

总计 1044

-rw-rw-r-- 1 grid grid 1048580 08-24 12:50 edits

-rw-rw-r-- 1 grid grid    4069 08-24 12:32 fsimage

-rw-rw-r-- 1 grid grid       8 08-24 12:32 fstime

-rw-rw-r-- 1 grid grid     101 08-24 12:32 VERSION

可见写数据的操作记录到了edits，注意：此时secondNamenode 并无任何变化

我们可以借用linux的stings查看edits中可读的内容：

[grid@h1 current]$ strings  edits

/user/grid/in/VERSION

1377319821684

1377319821684

67108864

grid

supergroup

$DFSClient_NONMAPREDUCE_-1780158786_1

192.168.76.2

/user/grid/in/VERSION

1377319821684

1377319821684

67108864

grid

supergroup

$DFSClient_NONMAPREDUCE_-1780158786_1

192.168.76.2

/user/grid/in/VERSION

1377319821684

1377319821684

67108864

grid

supergroup

有上我们很明显的意识到一个问题，如果不断有数据写入，edits启不是无限制的增长下去？hdfs是这样解决这个问题的：

运行secondNamenode,,为namenode内存中的元数据创建检查点

.

详细步骤如下：

1、secondNamenode请求namenode 停止使用edits,暂时将新的写操作记录到一个新的edits中

2、secondNamenode从namenode 获取fsimage和edits(采用http get)

3、secondNamenode将fsimage文件载入内存，逐一执行edits文件中的操作，创建新的fsimage文件。

4、secondNamenode将新的fsimage文件发送回namenode（使用http post)

5、namenode用接收到的fsimage替换旧的fsimage.

我们为了模拟检查点的发生，将fs.checkpoint.period由默认的3600秒改成120秒

重启hadoop，secendNamenode日志记录如下：

2013-08-24 12:56:16,457 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50090 webServer.getConnectors()[0].getLocalPort() returned 50090

2013-08-24 12:56:16,457 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50090

2013-08-24 12:56:16,457 INFO org.mortbay.log: jetty-6.1.26

2013-08-24 12:56:16,823 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50090

2013-08-24 12:56:16,823 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Web server init done

2013-08-24 12:56:16,823 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Secondary Web-server up at: 0.0.0.0:50090

2013-08-24 12:56:16,823 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:Checkpoint Period   :120 secs (2 min)

2013-08-24 12:56:16,824 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Log Size Trigger    :67108864 bytes (65536 KB)

两分钟过后secondNamenode开始运行检查点，日志记录如下：

2013-08-24 12:58:16,991 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection tohttp://0.0.0.0:50070/getimage?getimage=1

2013-08-24 12:58:17,115 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Downloaded file fsimage size 4164 bytes.

2013-08-24 12:58:17,115 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection tohttp://0.0.0.0:50070/getimage?getedit=1

2013-08-24 12:58:17,117 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Downloaded file edits size 805 bytes.

2013-08-24 12:58:17,119 INFO org.apache.hadoop.hdfs.util.GSet: Computing capacity for map BlocksMap

2013-08-24 12:58:17,119 INFO org.apache.hadoop.hdfs.util.GSet: VM type       = 32-bit

2013-08-24 12:58:17,119 INFO org.apache.hadoop.hdfs.util.GSet: 2.0% max memory = 1013645312

2013-08-24 12:58:17,119 INFO org.apache.hadoop.hdfs.util.GSet: capacity      = 2^22 = 4194304 entries

2013-08-24 12:58:17,119 INFO org.apache.hadoop.hdfs.util.GSet: recommended=4194304, actual=4194304

2013-08-24 12:58:17,193 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=grid

2013-08-24 12:58:17,193 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup

2013-08-24 12:58:17,193 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true

2013-08-24 12:58:17,200 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.block.invalidate.limit=100

2013-08-24 12:58:17,201 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)

2013-08-24 12:58:17,207 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: dfs.namenode.edits.toleration.length = -1

2013-08-24 12:58:17,207 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times

2013-08-24 12:58:17,411 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 34

2013-08-24 12:58:17,423 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 0

2013-08-24 12:58:17,423 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Start loading edits file /home/grid/hadoop-1.2.1/tmp/dfs/namesecondary/current/edits

2013-08-24 12:58:17,428 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: EOF of /home/grid/hadoop-1.2.1/tmp/dfs/namesecondary/current/edits, reached end of edit log Number of transactions found: 8.  Bytes read: 805

2013-08-24 12:58:17,428 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Edits file /home/grid/hadoop-1.2.1/tmp/dfs/namesecondary/current/edits of size 805 edits # 8 loaded in 0 seconds.

2013-08-24 12:58:17,428 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 0 Total time for transactions(ms): 0 Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0

2013-08-24 12:58:17,431 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: closing edit log: position=805, editlog=/home/grid/hadoop-1.2.1/tmp/dfs/namesecondary/current/edits

2013-08-24 12:58:17,431 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: close success: truncate to 805, editlog=/home/grid/hadoop-1.2.1/tmp/dfs/namesecondary/current/edits

2013-08-24 12:58:17,433 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file /home/grid/hadoop-1.2.1/tmp/dfs/namesecondary/current/fsimage of size 4164 bytes saved in 0 seconds.

2013-08-24 12:58:17,438 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: closing edit log: position=4, editlog=/home/grid/hadoop-1.2.1/tmp/dfs/namesecondary/current/edits

2013-08-24 12:58:17,438 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: close success: truncate to 4, editlog=/home/grid/hadoop-1.2.1/tmp/dfs/namesecondary/current/edits

2013-08-24 12:58:17,449 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Posted URL 0.0.0.0:50070putimage=1&port=50090&machine=0.0.0.0&token=-41:2085011414:0:1377320296000:1377320173851&newChecksum=96999e57430cf3ba909049d8a5cbf606

2013-08-24 12:58:17,449 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection to http://0.0.0.0:50070/getimage?putimage=1&port=50090&machine=0.0.0.0&token=-41:2085011414:0:1377320296000:1377320173851&newChecksum=96999e57430cf3ba909049d8a5cbf606

2013-08-24 12:58:17,541 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:Checkpoint done. New Image Size: 4164

检查点完成后，回看namenode edits的大小：

[grid@h1 current]$ ll

总计 20

-rw-rw-r-- 1 grid grid    4 08-24 12:58 edits

-rw-rw-r-- 1 grid grid 4164 08-24 12:58 fsimage

-rw-rw-r-- 1 grid grid    8 08-24 12:58 fstime

-rw-rw-r-- 1 grid grid  101 08-24 12:58 VERSION