hadoop 2.2搭建
来源:互联网 发布:52单片机蓝牙硬件电路 编辑:程序博客网 时间:2024/05/01 23:57
由于hadoop2.2.0是32位的,需要编译成64位的,现以做好编译成功的rar文件
http://download.csdn.net/detail/u011053107/7242945
http://download.csdn.net/detail/u011053107/7242935
另外提供32位hadoop下载地址wget http://mirrors.hust.edu.cn/apache/hadoop/common/stable2/hadoop-2.2.0.tar.gz
=====正式开始=====
tar -xzvfhadoop-2.2.0.tar.gz
cdhadoop-2.2.0
vi etc/hadoop/hadoop-env.sh
修改:
export JAVA_HOME=/usr/local/java/java-7-oracle/bin/java
vi etc/hadoop/slaves
修改localhost为host001
vi etc/hadoop/core-site.xml
增加:
<property>
<name>fs.defaultFS</name>
<value>hdfs://host001:9000</value>
</property>
cp etc/hadoop/mapred-site.xml.templateetc/hadoop/mapred-site.xml
vi etc/hadoop/mapred-site.xml
增加:
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.cluster.local.dir</name>
<value>/home/ysc/mapreduce/local</value>
</property>
vi etc/hadoop/yarn-site.xml
增加:
<property>
<name>yarn.resourcemanager.hostname</name>
<value>host001</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
vi etc/hadoop/hdfs-site.xml
增加:
<property>
<name>dfs.name.dir</name>
<value>/home/ysc/dfs/filesystem/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/ysc/dfs/filesystem/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
sudo vi /etc/proclusterIDfile
增加:
export HADOOP_PREFIX=/root/openstruct/hadoop-2.2.0
export HADOOP_COMMON_HOME=/root/openstruct/hadoop-2.2.0
export HADOOP_MAPRED_HOME=/root/openstruct/hadoop-2.2.0
export HADOOP_CONF_DIR=/root/openstruct/hadoop-2.2.0/etc/hadoop
export HADOOP_HDFS_HOME=/root/openstruct/hadoop-2.2.0
export HADOOP_YARN_HOME=/root/openstruct/hadoop-2.2.0
vi ./dfs/filesystem/data/current/VERSION
修改data节点的VERSION与name节点的clusterID值一致
clusterID=CID-889138e7-2a9c-4492-ac8c-63b339437563
source /etc/profile
格式化:
bin/hdfs namenode -format
启动dfs:
sbin/start-dfs.sh
启动yarn:
sbin/start-yarn.sh
启动historyserver:
sbin/mr-jobhistory-daemon.sh start historyserver
运行wordcount:
echo "APDPlat is a java open source project, Application Product Development Platform." > text1.txt
echo "Yang Shangchuan is the founder of APDPlat whichwon the "2013 Outstanding Open Source Project" award." > text2.txt
bin/hadoop fs -mkdir /input
bin/hadoop fs -put text1.txt /input
bin/hadoop fs -put text2.txt /input
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jarwordcount input output
访问管理页面:
ResourceManager:http://host001:8088
NodeManager:http://host001:8042
NameNode:http://host001:50070
NameNode:http://host001:50070
停止dfs:
sbin/stop-dfs.sh
停止yarn:
sbin/stop-yarn.sh
停止historyserver:
sbin/mr-jobhistory-daemon.sh stop historyserver
注意:要把这里的主机host001和用户ysc替换为你自己的,在/etc/hosts文件中要把127.0.0.1 localhost改为实际IP地址 host001
如果有需要在windows访问本发布管理平台,需要将windows下C:\Windows\System32\drivers\etc的hosts文件新增内容为192.168.6.141 host001
- hadoop 2.2搭建
- Hadoop 2.2 分布式集群搭建
- Hadoop 2.2.0集群搭建
- Hadoop 2.2.0 集群搭建
- hadoop 搭建
- hadoop搭建
- hadoop搭建
- hadoop搭建
- Hadoop搭建
- hadoop搭建
- 搭建hadoop
- hadoop搭建
- Hadoop搭建
- Hadoop搭建
- hadoop搭建
- hadoop 搭建
- Hadoop搭建
- hadoop 搭建
- 人生如梦,我总失眠,人生如戏,我总笑场
- jQuery实现翻滚图像(附代码)
- HDU 4951 Multiplication table(找规律)
- ClassPathResource
- vs2010安装后 F6不能编译生成
- hadoop 2.2搭建
- vs2010中,遇到“活动语句标记已被删除,并且无法在当前源文件中重新映射。必须恢复更改,才能继续或重新启动调试回话。”报错
- java.sql.SQLException: ORA-03115: 不支持的网络数据类型或表示法
- 自定义带下划线的EditText/TextView
- 端口状态说明 LISTENING、ESTABLISHED、TIME_WAIT及CLOSE_WAIT
- texturepacker
- 20140826
- 外勤人员管理的重要性
- mysql更新语句中的safe_mode