Hadoop伪分布式安装二(MapReduce+Yarn)

上一篇我们已经介绍了HDFS的部署，接着我们进行Hadoop伪分布式MapReduce+Yarn的部署

1.Yarn部署

• 修改mapred-site.xml 文件

[hadoop@zydatahadoop001 hadoop]$ cd        /opt/software/hadoop/etc/hadoop/[hadoop@zydatahadoop001 hadoop]$ cp mapred-site.xml.template mapred-site.xml[hadoop@zydatahadoop001 hadoop]$ vi mapred-site.xml添加下面的内容在<configuration>中    <property>        <name>mapreduce.framework.name</name>        <value>yarn</value>    </property>

• 修改yarn-site.xml文件

[hadoop@zydatahadoop001 hadoop]$ vi yarn-site.xml添加下面的内容在<configuration>中    <property>        <name>yarn.nodemanager.aux-services</name>        <value>mapreduce_shuffle</value>    </property>

• 开启服务

[hadoop@zydatahadoop001 hadoop]$ cd /opt/software/hadoop[hadoop@zydatahadoop001 hadoop]$ sbin/start-yarn.shstarting yarn daemonsstarting resourcemanager, logging to /opt/software/hadoop-2.8.1/logs/yarn-hadoop-resourcemanager-zydatahadoop001.outzydatahadoop001: starting nodemanager, logging to /opt/software/hadoop-2.8.1/logs/yarn-hadoop-nodemanager-zydatahadoop001.out[hadoop@zydatahadoop001 hadoop]$ jps24439 ResourceManager24840 Jps24073 SecondaryNameNode24539 NodeManager23788 NameNode查看8088端口[hadoop@zydatahadoop001 hadoop]$ netstat -nlp|grep 8088(Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.)tcp        0      0 :::8088                     :::*                        LISTEN      24439/java   

web界面进行查看: http://自己的ip:8088/

2.MR Job测试

MapReduce: java代码写的，map(映射)函数和reduce(归约)函数

这里可能出现错误：

WARN hdfs.DataStreamer: DataStreamer Exceptionorg.apache.hadoop.ipc.RemoteException(java.io.IOException):

解决办法参考这篇博客

• 计算圆周率

[hadoop@zydatahadoop001 hadoop]$ bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar pi 5 10计算结果（选取的参数问题出现误差）：Job Finished in 749.765 secondsEstimated value of Pi is 3.28000000000000000000

3.停止Yarn

[hadoop@zydatahadoop001 hadoop]$ sbin/stop-yarn.sh stopping yarn daemonsstopping resourcemanagerzydatahadoop001: stopping nodemanagerzydatahadoop001: nodemanager did not stop gracefully after 5 seconds: killing with kill -9no proxyserver to stop