Hadoop secondarynamenode应用

本文参考：http://www.cnblogs.com/ggjucheng/archive/2012/04/18/2454693.html，本文使用的是伪分布式的hadoop集群，版本为1.0.4，主要的配置文件如下：

core-site.xml:

<!-- Put site-specific property overrides in this file. --><configuration><property><name>hadoop.tmp.dir</name><value>/home/hadoop/fs/tmp</value></property><property><name>fs.default.name</name><value>hdfs://hadoop:9000</value></property><property><name>fs.checkpoint.period</name><value>300</value></property><property><name>fs.checkpoint.size</name><value>67108864</value></property><property><name>fs.checkpoint.dir</name><value>/home/hadoop/checkpoint/namesecodary</value></property></configuration>

hdfs-site.xml:

<!-- Put site-specific property overrides in this file. --><configuration><property><name>dfs.replicatioin</name><value>1</value></property><property><name>dfs.data.dir</name><value>/home/hadoop/fs/data</value></property><property><name>dfs.name.dir</name><value>/home/hadoop/fs/name</value></property><property><name>dfs.back.http.address</name><value>hadoop:50070</value></property><property><name>dfs.permissions</name><value>false</value></property></configuration>

mapred-site.xml:

<!-- Put site-specific property overrides in this file. --><configuration><property><name>mapred.job.tracker</name><value>hadoop:9001</value></property></configuration>

1.首先输入jps，查看hadoop进程，如下：

hadoop@hadoop:~/hadoop-1.0.4/bin$ jps16018 TaskTracker15646 DataNode17250 NameNode15870 JobTracker17296 Jps15790 SecondaryNameNode

2.kill掉namenode的进程，继续查看hadoop进程，如下：

hadoop@hadoop:~/hadoop-1.0.4/bin$ kill 17250hadoop@hadoop:~/hadoop-1.0.4/bin$ jps16018 TaskTracker15646 DataNode17328 Jps15870 JobTracker15790 SecondaryNameNode

3. 删除dfs.name.dir的目录的全部内容；

4. 这里没有按照参考的博客那样把secondarynamenode的fs.checkpoint.dir的内容拷贝到namenode的dfs.name.dir下面，而是直接使用./hadoop namenode -importCheckPoint命令，这个命令应该已经含有了拷贝的命令了，出现下面的logs信息：

hadoop@hadoop:~/hadoop-1.0.4/bin$ ./hadoop namenode -importCheckPoint13/05/28 15:37:14 INFO namenode.NameNode: STARTUP_MSG: /************************************************************STARTUP_MSG: Starting NameNodeSTARTUP_MSG:   host = hadoop/192.168.128.138STARTUP_MSG:   args = [-importCheckPoint]STARTUP_MSG:   version = 1.0.4STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct  3 05:13:58 UTC 2012************************************************************/13/05/28 15:37:14 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties13/05/28 15:37:14 INFO impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.13/05/28 15:37:14 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).13/05/28 15:37:14 INFO impl.MetricsSystemImpl: NameNode metrics system started13/05/28 15:37:14 INFO impl.MetricsSourceAdapter: MBean for source ugi registered.13/05/28 15:37:14 WARN impl.MetricsSystemImpl: Source name ugi already exists!13/05/28 15:37:14 INFO impl.MetricsSourceAdapter: MBean for source jvm registered.13/05/28 15:37:14 INFO impl.MetricsSourceAdapter: MBean for source NameNode registered.13/05/28 15:37:14 INFO util.GSet: VM type       = 32-bit13/05/28 15:37:14 INFO util.GSet: 2% max memory = 19.33375 MB13/05/28 15:37:14 INFO util.GSet: capacity      = 2^22 = 4194304 entries13/05/28 15:37:14 INFO util.GSet: recommended=4194304, actual=419430413/05/28 15:37:14 INFO namenode.FSNamesystem: fsOwner=hadoop13/05/28 15:37:14 INFO namenode.FSNamesystem: supergroup=supergroup13/05/28 15:37:14 INFO namenode.FSNamesystem: isPermissionEnabled=false13/05/28 15:37:14 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=10013/05/28 15:37:14 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)13/05/28 15:37:15 INFO namenode.FSNamesystem: Registered FSNamesystemStateMBean and NameNodeMXBean13/05/28 15:37:15 INFO namenode.NameNode: Caching file names occuring more than 10 times 13/05/28 15:37:15 INFO common.Storage: Storage directory /home/hadoop/fs/name is not formatted.13/05/28 15:37:15 INFO common.Storage: Formatting ...13/05/28 15:37:15 INFO common.Storage: Number of files = 813/05/28 15:37:15 INFO common.Storage: Number of files under construction = 013/05/28 15:37:15 INFO common.Storage: Image file of size 755 loaded in 0 seconds.13/05/28 15:37:15 INFO common.Storage: Edits file /home/hadoop/checkpoint/namesecodary/current/edits of size 4 edits # 0 loaded in 0 seconds.13/05/28 15:37:15 INFO common.Storage: Image file of size 755 saved in 0 seconds.13/05/28 15:37:15 INFO common.Storage: Image file of size 755 saved in 0 seconds.13/05/28 15:37:15 INFO namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 13/05/28 15:37:15 INFO common.Storage: Image file of size 755 saved in 0 seconds.13/05/28 15:37:15 INFO common.Storage: Image file of size 755 saved in 0 seconds.13/05/28 15:37:15 INFO namenode.NameCache: initialized with 0 entries 0 lookups13/05/28 15:37:15 INFO namenode.FSNamesystem: Finished loading FSImage in 345 msecs13/05/28 15:37:15 INFO hdfs.StateChange: STATE* Safe mode ON. The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. Safe mode will be turned off automatically.13/05/28 15:37:15 INFO util.HostsFileReader: Refreshing hosts (include/exclude) list13/05/28 15:37:15 INFO impl.MetricsSourceAdapter: MBean for source FSNamesystemMetrics registered.13/05/28 15:37:15 INFO impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort9000 registered.13/05/28 15:37:15 INFO impl.MetricsSourceAdapter: MBean for source RpcActivityForPort9000 registered.13/05/28 15:37:15 INFO ipc.Server: Starting SocketReader13/05/28 15:37:15 INFO namenode.NameNode: Namenode up at: hadoop/192.168.128.138:900013/05/28 15:37:35 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog13/05/28 15:37:35 INFO http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)13/05/28 15:37:35 INFO http.HttpServer: dfs.webhdfs.enabled = false13/05/28 15:37:35 INFO http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 5007013/05/28 15:37:35 INFO http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 5007013/05/28 15:37:35 INFO http.HttpServer: Jetty bound to port 5007013/05/28 15:37:35 INFO mortbay.log: jetty-6.1.2613/05/28 15:37:35 INFO mortbay.log: Started SelectChannelConnector@0.0.0.0:5007013/05/28 15:37:35 INFO namenode.NameNode: Web-server up at: 0.0.0.0:5007013/05/28 15:37:35 INFO ipc.Server: IPC Server Responder: starting13/05/28 15:37:35 INFO ipc.Server: IPC Server listener on 9000: starting13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 0 on 9000: starting13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 1 on 9000: starting13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 2 on 9000: starting13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 3 on 9000: starting13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 4 on 9000: starting13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 5 on 9000: starting13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 6 on 9000: starting13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 7 on 9000: starting13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 8 on 9000: starting13/05/28 15:37:35 INFO ipc.Server: IPC Server handler 9 on 9000: starting13/05/28 15:37:35 INFO hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from 192.168.128.138:50010 storage DS-2106362203-192.168.128.138-50010-136822953664013/05/28 15:37:35 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.128.138:5001013/05/28 15:37:35 INFO hdfs.StateChange: STATE* Safe mode extension entered. The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe mode will be turned off automatically in 29 seconds.13/05/28 15:37:35 INFO hdfs.StateChange: *BLOCK* NameSystem.processReport: from 192.168.128.138:50010, blocks: 1, processing time: 10 msecs13/05/28 15:37:56 INFO hdfs.StateChange: STATE* Safe mode ON. The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe mode will be turned off automatically in 9 seconds.13/05/28 15:38:06 INFO namenode.FSNamesystem: Total number of blocks = 113/05/28 15:38:06 INFO namenode.FSNamesystem: Number of invalid blocks = 013/05/28 15:38:06 INFO namenode.FSNamesystem: Number of under-replicated blocks = 113/05/28 15:38:06 INFO namenode.FSNamesystem: Number of  over-replicated blocks = 013/05/28 15:38:06 INFO hdfs.StateChange: STATE* Safe mode termination scan for invalid, over- and under-replicated blocks completed in 56 msec13/05/28 15:38:06 INFO hdfs.StateChange: STATE* Leaving safe mode after 51 secs.13/05/28 15:38:06 INFO hdfs.StateChange: STATE* Safe mode is OFF.13/05/28 15:38:06 INFO hdfs.StateChange: STATE* Network topology has 1 racks and 1 datanodes13/05/28 15:38:06 INFO hdfs.StateChange: STATE* UnderReplicatedBlocks has 1 blocks13/05/28 15:38:06 INFO namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 1 msec13/05/28 15:38:06 INFO namenode.FSNamesystem: ReplicateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 1 msec processing time, 1 msec clock time, 1 cycles13/05/28 15:38:06 INFO namenode.FSNamesystem: InvalidateQueue QueueProcessingStatistics: First cycle completed 0 blocks in 0 msec13/05/28 15:38:06 INFO namenode.FSNamesystem: InvalidateQueue QueueProcessingStatistics: Queue flush completed 0 blocks in 0 msec processing time, 0 msec clock time, 1 cycles13/05/28 15:38:06 INFO namenode.FSNamesystem: Roll Edit Log from 192.168.128.13813/05/28 15:38:06 INFO namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 13/05/28 15:38:06 INFO namenode.FSNamesystem: Roll FSImage from 192.168.128.13813/05/28 15:38:06 INFO namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 1 SyncTimes(ms): 5 

5.在kill掉namenode后，50070界面就不能访问了，但是使用了4的命令后，这个界面又可以访问了。但是使用CTRL+c后50070页面又不可以访问了，这时使用命令./hadoop-daemon.sh  start namenode 后稍等片刻就又可以访问50070界面了。