ubuntu搭建分布式hadoop-2.6.0概略和错误
来源:互联网 发布:图解冒泡排序算法 编辑:程序博客网 时间:2024/06/06 16:28
详细配置:http://blog.csdn.net/ggz631047367/article/details/42426391
1.修改机器/etc/hostname分别为
master
slave1
slave2
重启
2.修改/etc/hosts,删除多余项
192.168.2.223 master
192.168.2.222 slave2
192.168.2.224 slave1
注释掉127.0.0.1 localhost回环地址,及其它127.0.0.1地址和包含[master|slave1|slave2]的项
3.添加hadoop:hadoop用户名和组
4.安装免登陆SSH
5.安装JDK
6.安装hadoop
7.注意删除hadoop目录下dfs/data、tmp目录的所有文件
出错:
2015-11-11 01:26:28,191 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to master/172.17.0.27:9000. Exiting.
java.io.IOException: Incompatible clusterIDs in /usr/local/hadoop-2.6.0/dfs/data: namenode clusterID = CID-8a0b27a3-b53c-4e97-81e4-6937db707fc9; datanode clusterID = CID-b948da8a-e05a-4c45-97ff-7c2446000fb0
解决:查看dfs/name/current/VERSION中的clusterID ,将其余节点都改为这个值。
2015-11-11 02:36:18,651 FATAL org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.getDatanode: Data node DatanodeRegistration(172.17.0.24, datanodeUuid=6892adc4-0d44-4124-ba2a-927eee507954, infoPort=50075, ipcPort=50020, storageInfo=lv=-56;cid=CID-9cb6ddf6-1e61-4b34-906b-690ede335e8e;nsid=1714433657;c=0) is attempting to report storage ID 6892adc4-0d44-4124-ba2a-927eee507954. Node 172.17.0.28:50010 is expected to serve this storage.
解决:
删除tmp文件夹里文件
删除dfs/data
bin/hadoop datanode -format
2015/11/11 03:26:03 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1447211927171_0001
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Cannot delete /user/root/grep-temp-909165866. Name node is in safe mode.
The reported blocks 27 needs additional 8 blocks to reach the threshold 0.9990 of total blocks 35.
The number of live datanodes 3 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.
解决:
yarn-site.xml添加:
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
强制关闭
bin/hadoop dfsadmin -safemode leave
用户可以通过dfsadmin -safemode value 来操作安全模式,参数value的说明如下:
enter - 进入安全模式
leave - 强制NameNode离开安全模式
get - 返回安全模式是否开启的信息
wait - 等待,一直到安全模式结束。
2015-11-11 01:56:55,795 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService
java.io.EOFException: End of File Exception between local host is: "slave2/172.17.0.25"; destination host is: "master":9000; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException
解决:重新格式化NameNode
MapReduce卡住:
解决方法:
删除/etc/hosts中多余的项,比如:
127.0.0.1 localhost
127.0.0.1 slave1
不能用127.0.0.1代替本机地址,对应的错误:
2013-10-15 09:52:31,351 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: sunliang/192.168.1.232:9000. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
2013-10-15 09:52:32,352 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: sunliang/192.168.1.232:9000. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
解决:
删除/etc/hosts中多余的项
删除所用的tmp文件夹,然后执行hadoop namenode -format 进行格式化,在重新启动start-all.sh
还有个问题就是有防火墙,关闭防火墙
0 0
- ubuntu搭建分布式hadoop-2.6.0概略和错误
- Hadoop伪分布式环境搭建+Ubuntu:16.04+hadoop-2.6.0
- ubuntu 搭建伪分布式hadoop
- Ubuntu Hadoop伪分布式搭建
- Ubuntu Hadoop 完全分布式搭建
- Ubuntu-14.04.5搭建Hadoop-2.9.0分布式集群环境
- 搭建hadoop-2.6.0分布式集群
- hadoop 2.6.0 伪分布式搭建
- Hadoop-2.6.0完全分布式搭建
- 分布式Hadoop的搭建(Ubuntu)
- Ubuntu上搭建Hadoop环境(伪分布式)
- ubuntu 12.10 hadoop全分布式环境搭建
- Ubuntu hadoop 伪分布式环境搭建步骤
- Ubuntu下搭建Hadoop伪分布式
- Ubuntu搭建hadoop集群(分布式)
- Hadoop 2.6.0集群搭建(伪分布式+分布式)
- Hadoop 2.6.4 分布式搭建
- Ubuntu和CentOS中分布式配置Hadoop-2.2.0
- 数据库课堂笔记
- 关于iOS中的通知中心(NSNotificationCenter)
- 深入理解计算机系统(6)
- 稀疏编码(Sparse Coding)(二)
- 无法读取项目文件,请确认 <Import> 声明中的路径正确,且磁盘上存在该文件
- ubuntu搭建分布式hadoop-2.6.0概略和错误
- iOS网络请求,iOS 9新增加了Session请求
- Sparsity稀疏编码(三)
- stm32发字符串的小细节
- 第二章 身份验证
- ios 使用CFStringTransform将汉字转为拼音
- 杭电2594
- 稀疏编码最优化解法
- 第一章 Shiro简介