hadoop:dfsadmin -report 全0
来源:互联网 发布:html5仿微信红包源码 编辑:程序博客网 时间:2024/04/26 01:05
今早一来,突然发现使用-put命令往HDFS里传数据传不上去了,抱一大堆错误,然后我使用bin/hadoop dfsadmin -report查看系统状态
admin@adw1:/home/admin/joe.wangh/hadoop-0.19.2>bin/hadoop dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: ?%
-------------------------------------------------
Datanodes available: 0 (0 total, 0 dead)
使用bin/stop-all.sh关闭HADOOP
admin@adw1:/home/admin/joe.wangh/hadoop-0.19.2>bin/stop-all.sh
stopping jobtracker
172.16.197.192: stopping tasktracker
172.16.197.193: stopping tasktracker
stopping namenode
172.16.197.193: no datanode to stop
172.16.197.192: no datanode to stop
172.16.197.191: stopping secondarynamenode
哦,看到了吧,发现datanode前面并没有启动起来。去DATANODE上查看一下日志
admin@adw2:/home/admin/joe.wangh/hadoop-0.19.2/logs>vi hadoop-admin-datanode-adw2.hst.ali.dw.alidc.net.log
************************************************************/
2010-07-21 10:12:11,987 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /home/admin/joe.wangh/hadoop/data/dfs.data.dir: namenode namespaceID = 898136669; datanode namespaceID = 2127444065
at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:233)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:148)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:288)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:206)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1239)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1194)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1202)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1324)
......
错误提示namespaceIDs不一致。
下面给出两种解决办法,我使用的是第二种。
Workaround 1: Start from scratch
I can testify that the following steps solve this error, but the side effects won't make you happy (me neither). The crude workaround I have found is to:
1. stop the cluster
2. delete the data directory on the problematic datanode: the directory is specified by dfs.data.dir in conf/hdfs-site.xml; if you followed this tutorial, the relevant directory is /usr/local/hadoop-datastore/hadoop-hadoop/dfs/data
3. reformat the namenode (NOTE: all HDFS data is lost during this process!)
4. restart the cluster
When deleting all the HDFS data and starting from scratch does not sound like a good idea (it might be ok during the initial setup/testing), you might give the second approach a try.
Workaround 2: Updating namespaceID of problematic datanodes
Big thanks to Jared Stehler for the following suggestion. I have not tested it myself yet, but feel free to try it out and send me your feedback. This workaround is "minimally invasive" as you only have to edit one file on the problematic datanodes:
1. stop the datanode
2. edit the value of namespaceID in <dfs.data.dir>/current/VERSION to match the value of the current namenode
3. restart the datanode
If you followed the instructions in my tutorials, the full path of the relevant file is /usr/local/hadoop-datastore/hadoop-hadoop/dfs/data/current/VERSION (background: dfs.data.dir is by default set to ${hadoop.tmp.dir}/dfs/data, and we set hadoop.tmp.dir to /usr/local/hadoop-datastore/hadoop-hadoop).
If you wonder how the contents of VERSION look like, here's one of mine:
#contents of <dfs.data.dir>/current/VERSION
namespaceID=393514426
storageID=DS-1706792599-10.10.10.1-50010-1204306713481
cTime=1215607609074
storageType=DATA_NODE
layoutVersion=-13
原因:每次namenode format会重新创建一个namenodeId,而tmp/dfs/data下包含了上次format下的id,namenode format清空了namenode下的数据,但是没有晴空datanode下的数据,导致启动时失败,所要做的就是每次fotmat前,清空tmp一下 的所有目录.
转自:http://blog.csdn.net/wanghai__/article/details/5752199
- hadoop:dfsadmin -report 全0
- hadoop dfsadmin -report failed
- 使用hadoop dfsadmin -report命令查看集群运行,结果全为0,解决方法
- hadoop dfsadmin -report 命令详解
- hadoop dfsadmin -report命令详细信息
- hadoop dfsadmin
- hadoop dfsadmin
- dfsadmin -report结果分析
- sysctl命名、swappiness、dfsadmin -report
- hadoop dfsadmin -safemode 详解
- hadoop hdfs dfsadmin 介绍
- Hadoop管理命令 dfsadmin
- 调试dfsadmin里的report查询机制
- hdfs dfsadmin -report 查看集群节点
- hadoop dfsadmin -safemode 命令详解
- hadoop dfsadmin -refreshNodes 命令详解
- dfsadmin
- dfsadmin
- STL 之堆算法
- STL 之非修改算法
- android实现组件抖动
- JQuery框架的AJAX实例
- 了解黑客的关键工具---揭开Shellcode的神秘面纱
- hadoop:dfsadmin -report 全0
- session超时跳出iframe并跳到登陆页面
- Android:源码环境下移植第三方的apk内置到ROM(System Image)中
- 数字图像处理中的一些插值算法
- Android SurfaceView学习笔记
- android 通过流获取bitmap
- linux环境下通过nginx实现tomcat集群
- 文件夹遍历和防止程序启动两次
- 程序员应该要读的书