hadoop1.0.0 安装记录
来源:互联网 发布:个人网站源码带后台 编辑:程序博客网 时间:2024/05/22 15:15
操作系统为centos5.4(已经建立几个节点间的信任关系)
一,安装java
1,下载java (以下为下载在/work目录下操作)
wgethttp://download.oracle.com/otn-pub/java/jdk/7u2-b13/jdk-7u2-linux-i586.tar.gz
2,解压下载文件并改名
tar-zxvf jdk-7u2-linux-i586.tar.gz
mvjdk1.7.0_02 java
rmjdk-7u2-linux-i586.tar.gz
3,在/etc/profile中加入以下语句:
exportJAVA_HOME=/work/java
exportJRE_HOME=$JAVA_HOME/jre
exportPATH=$PATH:$JAVA_HOME/bin
二,安装hadoop
1, 下载hadoop压缩包(下载在/work目录下)
wget http://mirror.bit.edu.cn/apache//hadoop/common/hadoop-1.0.0/hadoop-1.0.0.tar.gz
2, 解压压缩包并改名
tar -zxvf hadoop-1.0.0.tar.gz
mv hadoop-1.0.0 hadoop
rm hadoop-1.0.0.tar.gz
3, 修改/etc/profile至
exportJAVA_HOME=/work/java
exportJRE_HOME=$JAVA_HOME/jre
exportHADOOP_HOME=/work/hadoop
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin
三,配置hadoop
1, 配置conf/hadoop-env.sh
export JAVA_HOME=/work/java
export HADOOP_HEAPSIZE=2000
2, 配置conf/core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://da-free-test1:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/work/hadoopneed/tmp</value>
</property>
<property>
<name>dfs.hosts.exclude</name>
<value>/work/hadoop/conf/dfs.hosts.exclude</value>
</property>
</configuration>
3, 配置hdfs-site.xml
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/work/hadoopneed/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/work/hadoopneed/data/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>30</value>
</property>
<property>
<name>dfs.datanode.handler.count</name>
<value>5</value>
</property>
<property>
<name>dfs.datanode.du.reserved</name>
<value>10737418240</value>
</property>
<property>
<name>dfs.block.size</name>
<value>134217728</value>
</property>
</configuration>
4, 配置mapred-site.xml
<property>
<name>mapred.job.tracker</name>
<value>da-free-test1:9001/</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/work/hadoopneed/mapred/local</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/tmp/hadoop/mapred/system</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx512</value>
<final>true</final>
</property>
<property>
<name>mapred.job.tracker.handler.count</name>
<value>30</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>100</value>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>12</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>63</value>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>6</value>
</property>
5, 配置masters
da-free-test1
6, 配置slaves
da-free-test2
da-free-test3
da-free-test4
四,其他节点的安装
1, 将hadoo和java目录拷贝到其他三个节点对应目录下
scp -rhadoop da-free-test2:/work
scp -rhadoop da-free-test3:/work
scp -rhadoop da-free-test4:/work
scp -rjava da-free-test2:/work
scp -rjava da-free-test3:/work
scp -r java da-free-test4:/work
2, 修改三个节点的/etc/profile,加入以下语句并执行一遍。
exportJAVA_HOME=/work/java
exportJRE_HOME=$JAVA_HOME/jre
exportHADOOP_HOME=/work/hadoop
exportPATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin
至此,算是安装完成,之后遇到的都当问题来处理。
六,格式化文件系统
1, 遇到问题:
[root@da-free-test1bin]# ./hadoop namenode -format
Warning:$HADOOP_HOME is deprecated.
Error: dlfailure on line 875
Error:failed /work/java/jre/lib/i386/server/libjvm.so, because/work/java/jre/lib/i386/server/libjvm.so: cannot restore segment prot afterreloc: Permission denied
Error: dlfailure on line 875
Error: failed /work/java/jre/lib/i386/server/libjvm.so,because /work/java/jre/lib/i386/server/libjvm.so: cannot restore segment protafter reloc: Permission denied
解决方法:关闭selinux:
修改/etc/selinux/config
SELINUX=disabled
更改其他三个节点,并重启系统。
解决报警
Warning: $HADOOP_HOME is deprecated.
将刚才添加到/etc/profile中的关于$HADOOP_HOME的删除并重新登录。
2, 成功格式化
[root@da-free-test1~]# hadoop namenode -format
12/02/0812:01:21 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG:Starting NameNode
STARTUP_MSG: host = da-free-test1/172.16.18.202
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.0
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0-r 1214675; compiled by 'hortonfo' on Thu Dec 15 16:36:35 UTC 2011
************************************************************/
12/02/0812:01:21 INFO util.GSet: VM type =32-bit
12/02/0812:01:21 INFO util.GSet: 2% max memory = 35.55625 MB
12/02/0812:01:21 INFO util.GSet: capacity =2^23 = 8388608 entries
12/02/0812:01:21 INFO util.GSet: recommended=8388608, actual=8388608
12/02/0812:01:21 INFO namenode.FSNamesystem: fsOwner=root
12/02/0812:01:21 INFO namenode.FSNamesystem: supergroup=supergroup
12/02/0812:01:21 INFO namenode.FSNamesystem: isPermissionEnabled=true
12/02/0812:01:21 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
12/02/0812:01:21 INFO namenode.FSNamesystem: isAccessTokenEnabled=falseaccessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
12/02/0812:01:21 INFO namenode.NameNode: Caching file names occuring more than 10 times
12/02/0812:01:22 INFO common.Storage: Image file of size 110 saved in 0 seconds.
12/02/0812:01:22 INFO common.Storage: Storage directory /work/hadoopneed/name has beensuccessfully formatted.
12/02/0812:01:22 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG:Shutting down NameNode at da-free-test1/172.16.18.202
************************************************************/
七,启动hadoop
1, 启动hadoop日志报错
./start-all.sh
WARN org.apache.hadoop.hdfs.DFSClient: DataStreamerException: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File/tmp/hadoop/mapred/system/jobtracker.info could only be replicated to 0 nodes,instead of 1
解决方法:关闭hadoop安全模式:hadoop dfsadmin -safemode leave(此时并未关闭hadoop)。
等待一会,hadoop自动恢复成功。
观察日志hadoop-root-jobtracker-da-free-test1.log,可以看到:
2012-02-08 12:14:07,804 INFOorg.apache.hadoop.ipc.Server: IPC Server Responder: starting
2012-02-08 12:14:07,804 INFOorg.apache.hadoop.ipc.Server: IPC Server listener on 9001: starting
2012-02-08 12:14:07,805 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 0 on 9001: starting
2012-02-08 12:14:07,805 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 1 on 9001: starting
2012-02-08 12:14:07,805 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001: starting
2012-02-08 12:14:07,805 INFO org.apache.hadoop.ipc.Server:IPC Server handler 3 on 9001: starting
2012-02-08 12:14:07,805 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 4 on 9001: starting
2012-02-08 12:14:07,805 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 5 on 9001: starting
2012-02-08 12:14:07,806 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 6 on 9001: starting
2012-02-08 12:14:07,806 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 7 on 9001: starting
2012-02-08 12:14:07,806 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 8 on 9001: starting
2012-02-08 12:14:07,806 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 9 on 9001: starting
2012-02-08 12:14:07,806 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 10 on 9001: starting
2012-02-08 12:14:07,806 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 11 on 9001: starting
2012-02-08 12:14:07,806 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 12 on 9001: starting
2012-02-08 12:14:07,806 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 13 on 9001: starting
2012-02-08 12:14:07,806 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 14 on 9001: starting
2012-02-08 12:14:07,807 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 15 on 9001: starting
2012-02-08 12:14:07,807 INFO org.apache.hadoop.ipc.Server:IPC Server handler 16 on 9001: starting
2012-02-08 12:14:07,807 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 17 on 9001: starting
2012-02-08 12:14:07,807 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 18 on 9001: starting
2012-02-08 12:14:07,807 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 19 on 9001: starting
2012-02-08 12:14:07,807 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 20 on 9001: starting
2012-02-08 12:14:07,807 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 21 on 9001: starting
2012-02-08 12:14:07,807 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 22 on 9001: starting
2012-02-08 12:14:07,808 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 23 on 9001: starting
2012-02-08 12:14:07,808 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 25 on 9001: starting
2012-02-08 12:14:07,808 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 26 on 9001: starting
2012-02-08 12:14:07,808 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 27 on 9001: starting
2012-02-08 12:14:07,808 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 28 on 9001: starting
2012-02-08 12:14:07,808 INFOorg.apache.hadoop.mapred.JobTracker: Starting RUNNING
2012-02-08 12:14:07,808 INFO org.apache.hadoop.ipc.Server:IPC Server handler 29 on 9001: starting
2012-02-08 12:14:07,808 INFOorg.apache.hadoop.ipc.Server: IPC Server handler 24 on 9001: starting
2012-02-08 12:14:12,623 INFOorg.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/da-free-test4
2012-02-08 12:14:12,625 INFOorg.apache.hadoop.mapred.JobTracker: Adding trackertracker_da-free-test4:da-free-test1/127.0.0.1:42182 to host da-free-test4
2012-02-08 12:14:12,743 INFOorg.apache.hadoop.net.NetworkTopology: Adding a new node:/default-rack/da-free-test3
2012-02-08 12:14:12,744 INFOorg.apache.hadoop.mapred.JobTracker: Adding trackertracker_da-free-test3:da-free-test1/127.0.0.1:53695 to host da-free-test3
2012-02-08 12:14:12,802 INFOorg.apache.hadoop.net.NetworkTopology: Adding a new node:/default-rack/da-free-test2
2012-02-08 12:14:12,802 INFOorg.apache.hadoop.mapred.JobTracker: Adding trackertracker_da-free-test2:da-free-test1/127.0.0.1:47259 to host da-free-test2
打开浏览器,输入http://172.16.18.202:50030,可以看到节点数量为3。
上传一个文件:dfs -put hadoop-root-namenode-da-free-test1.log /usr/testfile
查看上传的文件:hadoop dfs -ls /usr/
Found 1 items
-rw-r--r-- 3 root supergroup 845332012-02-08 12:20 /usr/testfile
查看http://172.16.18.202:50070,发现已用空间为396k。
暂时验证通过。
===========================
- hadoop1.0.0 安装记录
- hadoop1.0.0 安装记录
- Hadoop1安装
- CentOS6.5+hadoop1.2.1安装配置测试记录
- 在Mac OS上安装Hadoop1.0.0
- 安装hadoop1.0.3笔记
- hadoop1.1.1安装
- Hadoop1.0.4集群安装
- hadoop1.2.0安装
- hadoop1系列单机安装
- hadoop1.2.1安装
- hadoop1.2.1安装配置
- hadoop1.X安装
- Hadoop1完全分布式安装
- 安装配置hadoop1.2.1
- Hadoop1.1.2分布式安装
- Hadoop1.0.4集群安装
- centos6安装hadoop1.2.1
- 如何诊断高水位争用(enq: HW - contention)
- SQL SERVER 常用命令
- Android学习系列(12)--App列表之拖拽GridView
- 什么是shell 2 --百度百科
- 小波变换网文精粹:小波变换教程(五)
- hadoop1.0.0 安装记录
- MySQL 获得当前日期时间 函数
- Your project contains errors,please fix them before running your application
- Struts2知识点总结大全
- mysql 遇到的问题
- Xcode 4.1/4.2/4.2.1 免证书(iDP)开发+真机调试+生成IPA全攻略
- Android学习系列(15)--App列表之游标ListView(索引ListView)
- 不用IB 怎样给Control 注册事件
- jsp小区物业管理系统