3,数据挖掘环境搭建-Hadoop集群搭建与测试
来源:互联网 发布:mud游戏编程 编辑:程序博客网 时间:2024/06/05 07:00
安装hadoop-2.6.5.tar.gz
hdp10 namenode/datanode
hdp11 datanode
hdp12 datanode
0 安装前准备
修改hosts,修改主机名,关闭防火墙,关闭selinux,安装ssh,配置免密码,安装JDK
1,添加hadoop用户名
useradd hadoop
passwd hadoop
vi /etc/sudoers
root ALL=(ALL) ALLhadoop ALL=(ALL) ALL
2,修改/home/hadoop/apps/hadoop-2.6.5/etc/hadoop下的配置文件
2.1,修改hadoop-env.sh
vi hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_144
2.2 修改core-site.xml
vi core-site.xml
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hdp10:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/hdpdata</value> </property></configuration>
2.3,修改hdfs-site.xml
vi hdfs-site.xml
<configuration> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/hdpdata</value> </property> <property> <name>dfs.secondary.http.address</name> <value>hdp10:50070</value> </property> <property> <name>dfs.http.address</name> <value>hdp10:50090</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/home/hadoop/datanode</value> </property> <property> <name>dfs.namenode.data.dir</name> <value>/home/hadoop/namenode</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property></configuration>
2.4 修改mapred-site.xml
vi mapred-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property></configuration>
2.5,修改yarn-site.xml
vi yarn-site.xml
<configuration> <property> <name>yarn.resourcemanager.hostname</name> <value>hdp10</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property></configuration>
3,将hadoop文件添加到系统环境变量
vi /etc/profile
export HADOOP_HOME=/home/hadoop/apps/hadoop-2.6.5export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
4,修改slaves,设置datanode与nodemanager
vi /home/hadoop/apps/hadoop-2.6.5/etc/hadoop/slaves
hdp10
hdp11
hdp12
5,将/etc/profile和hadoop文件夹拷贝至hdp11,hdp12相应的位置
scp /etc/profile root@hdp11:/etc
scp /etc/profile root@hdp12:/etc
scp -r /home/hadoop/apps/hadoop-2.6.5 root@hdp11:/home/hadoop/apps
scp -r /home/hadoop/apps/hadoop-2.6.5 root@hdp12:/home/hadoop/apps
chown -R hadoop:hadoop /home/hadoop/apps/hadoop-2.6.5
6,去除HDFS WARN util.NativeCodeLoader
vi /home/hadoop/apps/hadoop-2.5.2/etc/hadoop/log4j.properties
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
7,格式化hadoop文件系统
hdfs namenode -format
8,启动hadoop
cd /home/hadoop/apps/hadoop-2.6.5/sbin
start-dfs.sh,启动HDFS
stop-dfs.sh,关闭HDFS
start-yarn.sh,启动YARN
stop-yarn.sh, 关闭YARN
9,查看启动状态
jps
10,范例
vi 1.txt
hello world hello counthello classhello rosehello jerryhello classhello rose
hadoop fs -mkdir -p /wordcount/input
hadoop fs -put 1.txt /wordcount/input
cd /home/hadoop/apps/hadoop-2.6.5/share/hadoop/mapreduce
hadoop jar hadoop-mapreduce-examples-2.6.5.jar wordcount /wordcount/input /wordcount/output0
- 3,数据挖掘环境搭建-Hadoop集群搭建与测试
- hadoop环境搭建与测试
- 5,数据挖掘环境搭建-Zookeeper集群搭建
- 6,数据挖掘环境搭建-Spark集群搭建
- 搭建Hadoop分布式集群------测试Hadoop分布式集群环境
- 环境搭建-Hadoop集群搭建
- Hadoop环境搭建-集群
- hadoop集群环境搭建
- 搭建hadoop集群环境
- Hadoop集群环境搭建
- 搭建hadoop集群环境
- 搭建hadoop集群环境
- 搭建hadoop集群环境
- 搭建hadoop集群环境
- 搭建hadoop集群环境
- hadoop集群环境搭建
- hadoop 集群环境搭建
- Hadoop集群环境搭建
- 报错解决问题
- 51talk-2 You look young
- javabean实体类对象转为Map类型对象的方法
- PHP函数闭包
- 【HDU
- 3,数据挖掘环境搭建-Hadoop集群搭建与测试
- vs2015常用快捷键总结
- 导入excel 读取单元格报错
- position中的absolute和relative
- Maven新建webapp项目index.jsp报错
- 批处理改ip
- Træfɪk
- HDU 5927 Auxiliary Set (树形)
- FTP服务基本原理与配置