centos 7下Hadoop 2.7.2 伪分布式安装
来源:互联网 发布:云计算阅读理解及答案 编辑:程序博客网 时间:2024/05/17 02:29
centos 7 下Hadoop 2.7.2 伪分布式安装,安装jdk,免密匙登录,配置mapreduce,配置YARN。详细步骤如下:
1.0 安装JDK
1.1 查看是否安装了openjdk
[lei@bogon ~]$ java -versionopenjdk version "1.8.0_65"OpenJDK Runtime Environment (build 1.8.0_65-b17)OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
1.2 查看openjdk源
[lei@bogon ~]$ rpm -qa | grep javajava-1.7.0-openjdk-1.7.0.91-2.6.2.3.el7.x86_64tzdata-java-2015g-1.el7.noarchpython-javapackages-3.4.1-11.el7.noarchjavapackages-tools-3.4.1-11.el7.noarchjava-1.8.0-openjdk-headless-1.8.0.65-3.b17.el7.x86_64java-1.8.0-openjdk-1.8.0.65-3.b17.el7.x86_64java-1.7.0-openjdk-headless-1.7.0.91-2.6.2.3.el7.x86_64
1.3 需root登录
[lei@bogon ~]$ rpm -e --nodeps tzdata-java-2015g-1.el7.noarch错误:can't create 事务 lock on /var/lib/rpm/.rpm.lock (权限不够)[lei@bogon ~]$ su密码:
1.4 依次删除OPENJDK
[root@bogon lei]# rpm -e --nodeps java-1.7.0-openjdk-1.7.0.91-2.6.2.3.el7.x86_64[root@bogon lei]# rpm -e --nodeps tzdata-java-2015g-1.el7.noarch[root@bogon lei]# rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.65-3.b17.el7.x86_64[root@bogon lei]# rpm -e --nodeps java-1.8.0-openjdk-1.8.0.65-3.b17.el7.x86_64[root@bogon lei]# rpm -e --nodeps java-1.7.0-openjdk-headless-1.7.0.91-2.6.2.3.el7.x86_64
1.5 将下载JDK后缀为.tar.gz解压缩至/opt路径下
[root@bogon lei]# tar -zxf/home/lei/桌面/jdk-8u77-linux-x64.tar.gz -C /opt
1.6 配置JDK环境变量
[root@bogon lei]# vim /etc/profile.d/java.sh
输入以下内容
JAVA_HOME=/opt/jdk1.8.0_77JRE_HOME=/opt/jdk1.8.0_77/jreCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATHPATH=$JAVA_HOME/bin:$PATH
1. 8使变量生效
source /etc/profile.d/java.sh
2.0 ssh 免密匙登录
是否可以免密匙登录
[root@bogon lei]# ssh localhostThe authenticity of host 'localhost (::1)' can't be established.ECDSA key fingerprint is 7b:10:e3:b5:ea:7d:29:be:77:83:1c:c0:1d:85:de:ba.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added 'localhost' (ECDSA) to the list of known hosts.[root@localhost's password:Last login: Sat Apr 2 22:32:44 2016
配置免密匙登录
[root@bogon ~]# ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsaGenerating public/private dsa key pair.Your identification has been saved in /root/.ssh/id_dsa.Your public key has been saved in /root/.ssh/id_dsa.pub.The key fingerprint is:e0:ba:3c:c5:73:dd:f0:39:51:ad:87:4b:dc:32:46:83 root@bogonThe key's randomart image is:+--[ DSA 1024]----+| . . || E + .|| . + = || . . . . O o|| .. S. + = = || .+ . . = . || .. o . || ... || o. |+-----------------+
[root@bogon ~]# cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys[root@bogon ~]# chmod 0600 ~/.ssh/authorized_keys
再次登录,即可免密匙
[root@bogon ~]# ssh localhostLast login: Sat Apr 2 22:51:41 2016 from localhost
3.0 安装Hadoop
3.0.1解压Hadoop至/opt路径下
tar -zxf /home/lei/桌面/hadoop-2.7.2.tar.gz -C /opt
3.0.2配置Hadoop环境变量
vim /etc/profile.d/hadoop.sh
输入以下内容
export HADOOP_HOME=/opt/hadoop-2.7.2export HADOOP_PREFIX=/opt/hadoop-2.7.2
使变量生效
source /etc/profile.d/hadoop.sh
3.0.3进入已经解压的Hadoop路径下
[root@bogon ~]# cd /opt/hadoop-2.7.2
3.0.4 修改hadoop-env.sh 增加JAVA_HOME路径
[root@bogon hadoop-2.7.2]# vim etc/hadoop/hadoop-env.sh
增加如下代码
export JAVA_HOME=/opt/
3.0.5 运行命令可看到Hadoop支持的命令
[root@bogon hadoop-2.7.2]# bin/hadoop
3.1 配置Mapreduce
3.1.1 编辑core-site.xml
[root@bogon hadoop-2.7.2]# vim etc/hadoop/core-site.xml<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property></configuration>
3.1.2 编辑hdfs-site.xml
[root@bogon hadoop-2.7.2]# vim etc/hadoop/hdfs-site.xml<configuration> <property> <name>dfs.replication</name> <value>1</value> </property></configuration>
3.1.3 格式化
[root@bogon hadoop-2.7.2]# bin/hdfs namenode -format
省略N行16/04/02 22:54:15 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted./************************************************************SHUTDOWN_MSG: Shutting down NameNode at bogon/221.192.153.42************************************************************/
3.1.4 开启mapreduce
[root@bogon hadoop-2.7.2]# sbin/start-dfs.sh
3.1.5 访问http://localhost:50070/
如图
3.1.6 运行mapreduce案例
[root@bogon hadoop-2.7.2]# bin/hdfs dfs -mkdir /user
[root@bogon hadoop-2.7.2]# bin/hdfs dfs -mkdir /user/lei
### 如果出现没有input错误[root@bogon hadoop-2.7.2]# bin/hdfs dfs -put etc/hadoop inputput: `input': No such file or directory### 手动创建[root@bogon hadoop-2.7.2]# bin/hadoop fs -mkdir -p input
[root@bogon hadoop-2.7.2]# bin/hdfs dfs -put etc/hadoop input
[root@bogon hadoop-2.7.2]# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar grep input output 'dfs[a-z.]+'
3.1.7 查看文件
[root@bogon hadoop-2.7.2]# bin/hadoop dfs -lsDEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it.Found 2 itemsdrwxr-xr-x - root supergroup 0 2016-04-02 23:39 inputdrwxr-xr-x - root supergroup 0 2016-04-02 23:43 output
3.1.8 停止mapreduce
[root@bogon hadoop-2.7.2]#sbin/start-dfs.sh
mapreduce配置完成
3.2 YARN配置
3.2.1 配置mapred-site.xml
[root@bogon hadoop-2.7.2]# mv etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml
[root@bogon hadoop-2.7.2]# vim etc/hadoop/mapred-site.xml<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property></configuration>
3.2.2配置yarn-site.xml
[root@bogon hadoop-2.7.2]# vim etc/hadoop/yarn-site.xml<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property></configuration>
[root@bogon hadoop-2.7.2]# sbin/start-yarn.sh
3.2.3 稍等片刻启动完毕后 打开http://localhost:8088/
如图
Hadoop伪分布式安装完成
3 0
- centos 7下Hadoop 2.7.2 伪分布式安装
- Centos 7中的HaDoop伪分布式安装
- CentOS下安装伪分布式Hadoop-1.2.1
- redHat6.4-x86_64下Hadoop 2.7.2 伪分布式安装
- Hadoop 2.7.1在centos 6.3下配置变量环境并安装配置伪分布式集群
- centos 6.3下安装Hadoop 2.7.1并配置伪分布式集群
- 在 CentOS 7.2 下安装 Hadoop 2.7.5 并搭建伪分布式环境的方法
- CentOS hadoop 伪分布式安装步骤
- ubuntu 下安装伪分布式 hadoop
- linux下安装hadoop伪分布式集群
- CentOS下Hadoop伪分布模式安装
- Hadoop 伪分布式安装
- hadoop 伪分布式安装
- Hadoop 伪分布式安装
- HADOOP伪分布式安装
- hadoop伪分布式安装
- Hadoop伪分布式安装
- Hadoop伪分布式安装
- 浅尝微信朋友圈的细节之美
- vim命令集及快捷键
- 使用BusyBox解决adb shell命令无法使用awk等命令的问题
- Bzoj1856:[Scoi2010]字符串:组合数学
- 使用blat实现在window系统下通过BAT批处理脚本发邮件功能
- centos 7下Hadoop 2.7.2 伪分布式安装
- github 网页版的使用
- hbase基本命令
- 神经网络机器翻译Neural Machine Translation(1): Encoder-Decoder Architecture
- iOS应用开发中的MD5加密的相关使用
- foreach循环在数组和集合中的使用
- 一个网关与多UI应用
- MMO游戏技能攻击区域的计算2--给地图划分格子
- 第2课:全球详细图解JVM内存三大核心区域及其JVM内存案例实战剖析