hadoop 之路我遇到的问题
来源:互联网 发布:mac csv 中文乱码 编辑:程序博客网 时间:2024/04/30 11:52
1、 Incompatible namespaceIDs 错误
-put命令往HDFS里传数据传不上去了,抱一大堆错误,然后我使用bin/hadoop dfsadmin -report查看系统状态
admin@adw1:/home/admin/joe.wangh/hadoop-0.19.2>bin/hadoop dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 0 (0 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 0 (0 KB)
DFS Used%: ?%
-------------------------------------------------
Datanodes available: 0 (0 total, 0 dead)
使用bin/stop-all.sh关闭HADOOP
admin@adw1:/home/admin/joe.wangh/hadoop-0.19.2>bin/stop-all.sh
stopping jobtracker
172.16.197.192: stopping tasktracker
172.16.197.193: stopping tasktracker
stopping namenode
172.16.197.193: no datanode to stop
172.16.197.192: no datanode to stop
172.16.197.191: stopping secondarynamenode
哦,看到了吧,发现datanode前面并没有启动起来。去DATANODE上查看一下日志
admin@adw2:/home/admin/joe.wangh/hadoop-0.19.2/logs>vi hadoop-admin-datanode-adw2.hst.ali.dw.alidc.net.log
************************************************************/
2010-07-21 10:12:11,987 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /home/admin/joe.wangh/hadoop/data/dfs.data.dir: namenode namespaceID = 898136669; datanode namespaceID = 2127444065
at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:233)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:148)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:288)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:206)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1239)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1194)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1202)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1324)
......
错误提示namespaceIDs不一致。
下面给出两种解决办法,我使用的是第二种。
Workaround 1: Start from scratch
I can testify that the following steps solve this error, but the side effects won't make you happy (me neither). The crude workaround I have found is to:
1. stop the cluster
2. delete the data directory on the problematic datanode: the directory is specified by dfs.data.dir in conf/hdfs-site.xml; if you followed this tutorial, the relevant directory is /usr/local/hadoop-datastore/hadoop-hadoop/dfs/data
3. reformat the namenode (NOTE: all HDFS data is lost during this process!)
4. restart the cluster
When deleting all the HDFS data and starting from scratch does not sound like a good idea (it might be ok during the initial setup/testing), you might give the second approach a try.
Workaround 2: Updating namespaceID of problematic datanodes
Big thanks to Jared Stehler for the following suggestion. I have not tested it myself yet, but feel free to try it out and send me your feedback. This workaround is "minimally invasive" as you only have to edit one file on the problematic datanodes:
1. stop the datanode
2. edit the value of namespaceID in <dfs.data.dir>/current/VERSION to match the value of the current namenode
3. restart the datanode
If you followed the instructions in my tutorials, the full path of the relevant file is /usr/local/hadoop-datastore/hadoop-hadoop/dfs/data/current/VERSION (background: dfs.data.dir is by default set to ${hadoop.tmp.dir}/dfs/data, and we set hadoop.tmp.dir to /usr/local/hadoop-datastore/hadoop-hadoop).
If you wonder how the contents of VERSION look like, here's one of mine:
#contents of <dfs.data.dir>/current/VERSION
namespaceID=393514426
storageID=DS-1706792599-10.10.10.1-50010-1204306713481
cTime=1215607609074
storageType=DATA_NODE
layoutVersion=-13
原因:每次namenode format会重新创建一个namenodeId,而tmp/dfs/data下包含了上次format下的id,namenode format清空了namenode下的数据,但是没有晴空datanode下的数据,导致启动时失败,所要做的就是每次fotmat前,清空tmp一下的所有目录.
- hadoop 之路我遇到的问题
- 我的小锦囊之Hadoop集群的启动命令以及经常遇到的小问题
- eclispe初次运行Hadoop时,我遇到的问题。
- hadoop遇到的问题
- 【Hadoop进阶之路】Hadoop部署过程中遇到的问题
- 我遇到的问题!!
- 我遇到的问题
- 我遇到的问题
- 运行Hadoop遇到的问题
- hadoop遇到的问题记录
- hadoop遇到的一些问题
- 初步 HADOOP 遇到的问题
- hadoop 程序遇到的问题
- hadoop 遇到的问题记录
- 初学hadoop遇到的问题
- Hadoop开发遇到的问题
- hadoop 应用遇到的问题
- Hadoop可能遇到的问题
- C Standard Library
- XML入门精解之DTD 文件格式定义(XML DTD)
- 微软提供免费手机与开发工具等吸引WebOS开发者
- centos下面安装火狐flash插件
- android系统开发(四)-触摸屏tslib移植(内核)和原理分析
- hadoop 之路我遇到的问题
- string.format() 或是.tostring(xxx)格式
- Yahoo为啥赚不到钱
- eclipse 启动 JVM terminated. exit code 1 错误解决办法
- 为爱感动
- VC导出CListCtrl数据到Excel两种方法
- emulator从eclipse启动出错非法关闭解决办法
- intent 用法
- java网络编程