Hadoop
来源：互联网 发布：java语言程序设计 一 编辑：程序博客网 时间：2024/06/11 05:36
系统版本
anliven@Ubuntu1604:~$ uname -aLinux Ubuntu1604 4.8.0-36-generic #36~16.04.1-Ubuntu SMP Sun Feb 5 09:39:57 UTC 2017 x86_64 x86_64 x86_64 GNU/Linuxanliven@Ubuntu1604:~$ anliven@Ubuntu1604:~$ cat /proc/versionLinux version 4.8.0-36-generic (buildd@lgw01-18) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #36~16.04.1-Ubuntu SMP Sun Feb 5 09:39:57 UTC 2017anliven@Ubuntu1604:~$ anliven@Ubuntu1604:~$ lsb_release -aNo LSB modules are available.Distributor ID: UbuntuDescription: Ubuntu 16.04.2 LTSRelease: 16.04Codename: xenialanliven@Ubuntu1604:~$ 创建hadoop用户
anliven@Ubuntu1604:~$ sudo useradd -m hadoop -s /bin/bashanliven@Ubuntu1604:~$ sudo passwd hadoop输入新的 UNIX 密码： 重新输入新的 UNIX 密码： passwd：已成功更新密码anliven@Ubuntu1604:~$ anliven@Ubuntu1604:~$ sudo adduser hadoop sudo正在添加用户"hadoop"到"sudo"组...正在将用户“hadoop”加入到“sudo”组中完成。anliven@Ubuntu1604:~$ 更新apt及安装vim
hadoop@Ubuntu1604:~$ sudo apt-get update命中:1 http://mirrors.aliyun.com/ubuntu xenial InRelease命中:2 http://mirrors.aliyun.com/ubuntu xenial-updates InRelease命中:3 http://mirrors.aliyun.com/ubuntu xenial-backports InRelease命中:4 http://mirrors.aliyun.com/ubuntu xenial-security InRelease正在读取软件包列表... 完成 hadoop@Ubuntu1604:~$ hadoop@Ubuntu1604:~$ sudo apt-get install vim正在读取软件包列表... 完成正在分析软件包的依赖关系树 正在读取状态信息... 完成 vim 已经是最新版 (2:7.4.1689-3ubuntu1.2)。升级了 0 个软件包，新安装了 0 个软件包，要卸载 0 个软件包，有 50 个软件包未被升级。hadoop@Ubuntu1604:~$ 配置SSH免密码登录
hadoop@Ubuntu1604:~$ sudo apt-get install openssh-server正在读取软件包列表... 完成正在分析软件包的依赖关系树 正在读取状态信息... 完成 openssh-server 已经是最新版 (1:7.2p2-4ubuntu2.1)。升级了 0 个软件包，新安装了 0 个软件包，要卸载 0 个软件包，有 50 个软件包未被升级。hadoop@Ubuntu1604:~$ hadoop@Ubuntu1604:~$ cd ~hadoop@Ubuntu1604:~$ mkdir .sshhadoop@Ubuntu1604:~$ cd .sshhadoop@Ubuntu1604:~/.ssh$ ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa.Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.The key fingerprint is:SHA256:DzjVWgTQB5I1JGRBmWi6gVHJ03V4WnJZEdojtbou0DM hadoop@Ubuntu1604The key's randomart image is:+---[RSA 2048]----+| o.o =X@B=*o ||. + +.*+*B.. || o + *+.* ||. o .o = . || o .o S || . . E. + || . o. . || .. || .. |+----[SHA256]-----+hadoop@Ubuntu1604:~/.ssh$ hadoop@Ubuntu1604:~/.ssh$ cat id_rsa.pub >> authorized_keyshadoop@Ubuntu1604:~/.ssh$ ls -l总用量 12-rw-rw-r-- 1 hadoop hadoop 399 4月 27 07:33 authorized_keys-rw------- 1 hadoop hadoop 1679 4月 27 07:32 id_rsa-rw-r--r-- 1 hadoop hadoop 399 4月 27 07:32 id_rsa.pubhadoop@Ubuntu1604:~/.ssh$ hadoop@Ubuntu1604:~/.ssh$ cd hadoop@Ubuntu1604:~$ hadoop@Ubuntu1604:~$ ssh localhostThe authenticity of host 'localhost (127.0.0.1)' can't be established.ECDSA key fingerprint is SHA256:fZ7fAvnnFk0/Imkn0YPdc2Gzxnfr0IJGSRb1swbm7oU.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added 'localhost' (ECDSA) to the list of known hosts.Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.8.0-36-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage44 个可升级软件包。0 个安全更新。*** 需要重启系统 ***Last login: Thu Apr 27 07:25:26 2017 from 192.168.16.1hadoop@Ubuntu1604:~$ hadoop@Ubuntu1604:~$ exit注销Connection to localhost closed.hadoop@Ubuntu1604:~$ 安装Java
hadoop@Ubuntu1604:~$ dpkg -l |grep jdkhadoop@Ubuntu1604:~$ hadoop@Ubuntu1604:~$ sudo apt-get install openjdk-8-jre openjdk-8-jdk正在读取软件包列表... 完成正在分析软件包的依赖关系树 正在读取状态信息... 完成 将会同时安装下列软件：..................done.正在处理用于 libc-bin (2.23-0ubuntu7) 的触发器 ...正在处理用于 ca-certificates (20160104ubuntu1) 的触发器 ...Updating certificates in /etc/ssl/certs...0 added, 0 removed; done.Running hooks in /etc/ca-certificates/update.d...done.done.hadoop@Ubuntu1604:~$ hadoop@Ubuntu1604:~$ dpkg -l |grep jdkii openjdk-8-jdk:amd64 8u121-b13-0ubuntu1.16.04.2 amd64 OpenJDK Development Kit (JDK)ii openjdk-8-jdk-headless:amd64 8u121-b13-0ubuntu1.16.04.2 amd64 OpenJDK Development Kit (JDK) (headless)ii openjdk-8-jre:amd64 8u121-b13-0ubuntu1.16.04.2 amd64 OpenJDK Java runtime, using Hotspot JITii openjdk-8-jre-headless:amd64 8u121-b13-0ubuntu1.16.04.2 amd64 OpenJDK Java runtime, using Hotspot JIT (headless)hadoop@Ubuntu1604:~$ hadoop@Ubuntu1604:~$ dpkg -L openjdk-8-jdk | grep '/bin$'/usr/lib/jvm/java-8-openjdk-amd64/binhadoop@Ubuntu1604:~$ hadoop@Ubuntu1604:~$ vim ~/.bashrchadoop@Ubuntu1604:~$ hadoop@Ubuntu1604:~$ head ~/.bashrc |grep javaexport JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64"hadoop@Ubuntu1604:~$ hadoop@Ubuntu1604:~$ source ~/.bashrchadoop@Ubuntu1604:~$ hadoop@Ubuntu1604:~$ echo $JAVA_HOME/usr/lib/jvm/java-8-openjdk-amd64hadoop@Ubuntu1604:~$ hadoop@Ubuntu1604:~$ java -versionopenjdk version "1.8.0_121"OpenJDK Runtime Environment (build 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13)OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)hadoop@Ubuntu1604:~$ 安装Hadoop
hadoop@Ubuntu1604:~$ sudo tar -zxf ~/hadoop-2.8.0.tar.gz -C /usr/local[sudo] hadoop 的密码： hadoop@Ubuntu1604:~$ cd /usr/localhadoop@Ubuntu1604:/usr/local$ sudo mv ./hadoop-2.8.0/ ./hadoophadoop@Ubuntu1604:/usr/local$ sudo chown -R hadoop ./hadoophadoop@Ubuntu1604:/usr/local$ ls -l |grep hadoopdrwxr-xr-x 9 hadoop dialout 4096 3月 17 13:31 hadoophadoop@Ubuntu1604:/usr/local$ cd ./hadoophadoop@Ubuntu1604:/usr/local/hadoop$ ls -l总用量 148drwxr-xr-x 2 hadoop dialout 4096 3月 17 13:31 bindrwxr-xr-x 3 hadoop dialout 4096 3月 17 13:31 etcdrwxr-xr-x 2 hadoop dialout 4096 3月 17 13:31 includedrwxr-xr-x 3 hadoop dialout 4096 3月 17 13:31 libdrwxr-xr-x 2 hadoop dialout 4096 3月 17 13:31 libexec-rw-r--r-- 1 hadoop dialout 99253 3月 17 13:31 LICENSE.txt-rw-r--r-- 1 hadoop dialout 15915 3月 17 13:31 NOTICE.txt-rw-r--r-- 1 hadoop dialout 1366 3月 17 13:31 README.txtdrwxr-xr-x 2 hadoop dialout 4096 3月 17 13:31 sbindrwxr-xr-x 4 hadoop dialout 4096 3月 17 13:31 sharehadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hadoop versionHadoop 2.8.0Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 91f2b7a13d1e97be65db92ddabc627cc29ac0009Compiled by jdu on 2017-03-17T04:12ZCompiled with protoc 2.5.0From source with checksum 60125541c2b3e266cbf3becc5bda666This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.8.0.jarhadoop@Ubuntu1604:/usr/local/hadoop$ Hadoop伪分布式配置
Hadoop可以伪分布式的方式在单节点上运行，读取HDFS中的文件。此节点既作为 NameNode 也作为 DataNode。
在Hadoop伪分布式配置情况下，删除core-site.xml的配置项，可以从伪分布式模式切换回非分布式模式。
修改配置文件
hadoop@Ubuntu1604:~$ cd /usr/local/hadoop/etc/hadoophadoop@Ubuntu1604:/usr/local/hadoop/etc/hadoop$ vim core-site.xml hadoop@Ubuntu1604:/usr/local/hadoop/etc/hadoop$ cat core-site.xml <?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><configuration> <property> <name>hadoop.tmp.dir</name> <value>file:/usr/local/hadoop/tmp</value> <description>Abase for other temporary directories.</description> </property> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property></configuration>hadoop@Ubuntu1604:/usr/local/hadoop/etc/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop/etc/hadoop$ vim hdfs-site.xmlhadoop@Ubuntu1604:/usr/local/hadoop/etc/hadoop$ cat hdfs-site.xml <?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/usr/local/hadoop/tmp/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/usr/local/hadoop/tmp/dfs/data</value> </property></configuration>hadoop@Ubuntu1604:/usr/local/hadoop/etc/hadoop$ 格式化NameNode
hadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hdfs namenode -format17/04/27 23:39:01 INFO namenode.NameNode: STARTUP_MSG: /************************************************************STARTUP_MSG: Starting NameNodeSTARTUP_MSG: user = hadoopSTARTUP_MSG: host = Ubuntu1604/127.0.1.1STARTUP_MSG: args = [-format]STARTUP_MSG: version = 2.8.0..................17/04/27 23:39:02 INFO namenode.FSImage: Allocated new BlockPoolId: BP-806199003-127.0.1.1-149330754208617/04/27 23:39:02 INFO common.Storage: Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.17/04/27 23:39:02 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression17/04/27 23:39:02 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 323 bytes saved in 0 seconds.17/04/27 23:39:02 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 017/04/27 23:39:02 INFO util.ExitUtil: Exiting with status 017/04/27 23:39:02 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************SHUTDOWN_MSG: Shutting down NameNode at Ubuntu1604/127.0.1.1************************************************************/hadoop@Ubuntu1604:/usr/local/hadoop$ 启动NameNode和DataNode进程
hadoop@Ubuntu1604:/usr/local/hadoop$ ./sbin/start-dfs.shStarting namenodes on [localhost]localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-Ubuntu1604.outlocalhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-Ubuntu1604.outStarting secondary namenodes [0.0.0.0]The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.ECDSA key fingerprint is SHA256:fZ7fAvnnFk0/Imkn0YPdc2Gzxnfr0IJGSRb1swbm7oU.Are you sure you want to continue connecting (yes/no)? yes0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-Ubuntu1604.outhadoop@Ubuntu1604:/usr/local/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop$ jps1908 Jps1576 DataNode1467 NameNode1791 SecondaryNameNodehadoop@Ubuntu1604:/usr/local/hadoop$ 访问Web界面
hadoop@Ubuntu1604:/usr/local/hadoop$ ip addr show enp0s32: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:02:49:c1 brd ff:ff:ff:ff:ff:ff inet 192.168.16.100/24 brd 192.168.16.255 scope global enp0s3 valid_lft forever preferred_lft forever inet6 fe80::a00:27ff:fe02:49c1/64 scope link valid_lft forever preferred_lft foreverhadoop@Ubuntu1604:/usr/local/hadoop$ 访问Web界面http://192.168.16.100:50070,可以查看NameNode/Datanode信息和HDFS中的文件
运行Hadoop伪分布式实例
hadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hdfs dfs -mkdir -p /user/hadoophadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hdfs dfs -mkdir inputhadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hdfs dfs -put ./etc/hadoop/*.xml inputhadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hdfs dfs -lsFound 1 itemsdrwxr-xr-x - hadoop supergroup 0 2017-04-29 07:42 inputhadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hdfs dfs -ls inputFound 8 items-rw-r--r-- 1 hadoop supergroup 4942 2017-04-29 07:42 input/capacity-scheduler.xml-rw-r--r-- 1 hadoop supergroup 1111 2017-04-29 07:42 input/core-site.xml-rw-r--r-- 1 hadoop supergroup 9683 2017-04-29 07:42 input/hadoop-policy.xml-rw-r--r-- 1 hadoop supergroup 1181 2017-04-29 07:42 input/hdfs-site.xml-rw-r--r-- 1 hadoop supergroup 620 2017-04-29 07:42 input/httpfs-site.xml-rw-r--r-- 1 hadoop supergroup 3518 2017-04-29 07:42 input/kms-acls.xml-rw-r--r-- 1 hadoop supergroup 5546 2017-04-29 07:42 input/kms-site.xml-rw-r--r-- 1 hadoop supergroup 690 2017-04-29 07:42 input/yarn-site.xmlhadoop@Ubuntu1604:/usr/local/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.0.jar grep input output 'dfs[a-z.]+'17/04/29 07:43:54 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id..................17/04/29 07:43:58 INFO mapreduce.Job: map 100% reduce 100%17/04/29 07:43:58 INFO mapreduce.Job: Job job_local329465708_0002 completed successfully17/04/29 07:43:58 INFO mapreduce.Job: Counters: 35 File System Counters FILE: Number of bytes read=1222362 FILE: Number of bytes written=2503241 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=55020 HDFS: Number of bytes written=515 HDFS: Number of read operations=67 HDFS: Number of large read operations=0 HDFS: Number of write operations=16 Map-Reduce Framework Map input records=4 Map output records=4 Map output bytes=101 Map output materialized bytes=115 Input split bytes=132 Combine input records=0 Combine output records=0 Reduce input groups=1 Reduce shuffle bytes=115 Reduce input records=4 Reduce output records=4 Spilled Records=8 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=0 Total committed heap usage (bytes)=1054867456 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=219 File Output Format Counters Bytes Written=77hadoop@Ubuntu1604:/usr/local/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hdfs dfs -lsFound 2 itemsdrwxr-xr-x - hadoop supergroup 0 2017-04-29 07:42 inputdrwxr-xr-x - hadoop supergroup 0 2017-04-29 07:43 outputhadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hdfs dfs -ls outputFound 2 items-rw-r--r-- 1 hadoop supergroup 0 2017-04-29 07:43 output/_SUCCESS-rw-r--r-- 1 hadoop supergroup 77 2017-04-29 07:43 output/part-r-00000hadoop@Ubuntu1604:/usr/local/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hdfs dfs -cat output/*1 dfsadmin1 dfs.replication1 dfs.namenode.name.dir1 dfs.datanode.data.dirhadoop@Ubuntu1604:/usr/local/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop$ ls -l ./outputls: 无法访问'./output': 没有那个文件或目录hadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hdfs dfs -get output ./outputhadoop@Ubuntu1604:/usr/local/hadoop$ cat ./output/*1 dfsadmin1 dfs.replication1 dfs.namenode.name.dir1 dfs.datanode.data.dirhadoop@Ubuntu1604:/usr/local/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hdfs dfs -rm -r outputDeleted outputhadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hdfs dfs -lsFound 1 itemsdrwxr-xr-x - hadoop supergroup 0 2017-04-29 07:42 inputhadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.0.jar grep input output 'dfs[a-z.]+'17/04/29 07:48:40 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id..................hadoop@Ubuntu1604:/usr/local/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hdfs dfs -lsFound 2 itemsdrwxr-xr-x - hadoop supergroup 0 2017-04-29 07:42 inputdrwxr-xr-x - hadoop supergroup 0 2017-04-29 07:48 outputhadoop@Ubuntu1604:/usr/local/hadoop$ ./bin/hdfs dfs -cat output/*1 dfsadmin1 dfs.replication1 dfs.namenode.name.dir1 dfs.datanode.data.dirhadoop@Ubuntu1604:/usr/local/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop$ ./sbin/stop-dfs.shStopping namenodes on [localhost]localhost: stopping namenodelocalhost: stopping datanodeStopping secondary namenodes [0.0.0.0]0.0.0.0: stopping secondarynamenodehadoop@Ubuntu1604:/usr/local/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop$ jps3807 Jpshadoop@Ubuntu1604:/usr/local/hadoop$ 特别注意：Hadoop运行程序时，输出目录不能存在，否则会出错。再次执行前，必须删除 output 文件夹：./bin/hdfs dfs -rm -r output。
YARN
修改配置文件mapred-site.xml
hadoop@Ubuntu1604:/usr/local/hadoop/etc/hadoop$ pwd/usr/local/hadoop/etc/hadoophadoop@Ubuntu1604:/usr/local/hadoop/etc/hadoop$ mv mapred-site.xml.template mapred-site.xmlhadoop@Ubuntu1604:/usr/local/hadoop/etc/hadoop$ vim mapred-site.xmlhadoop@Ubuntu1604:/usr/local/hadoop/etc/hadoop$ cat mapred-site.xml<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property></configuration>hadoop@Ubuntu1604:/usr/local/hadoop/etc/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop/etc/hadoop$ vim yarn-site.xmlhadoop@Ubuntu1604:/usr/local/hadoop/etc/hadoop$ cat yarn-site.xml <?xml version="1.0"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property></configuration>hadoop@Ubuntu1604:/usr/local/hadoop/etc/hadoop$ 如果不想启动YARN，务必将配置文件 mapred-site.xml改为原名称mapred-site.xml.template，否则将很可能会引起程序异常。
启动YARN
hadoop@Ubuntu1604:/usr/local/hadoop$ pwd/usr/local/hadoophadoop@Ubuntu1604:/usr/local/hadoop$ jps5774 Jpshadoop@Ubuntu1604:/usr/local/hadoop$ ./sbin/start-dfs.shStarting namenodes on [localhost]localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-Ubuntu1604.outlocalhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-Ubuntu1604.outStarting secondary namenodes [0.0.0.0]0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-Ubuntu1604.outhadoop@Ubuntu1604:/usr/local/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop$ jps6034 DataNode6373 Jps5915 NameNode6221 SecondaryNameNodehadoop@Ubuntu1604:/usr/local/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop$ ./sbin/start-yarn.shstarting yarn daemonsstarting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-resourcemanager-Ubuntu1604.outlocalhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-Ubuntu1604.outhadoop@Ubuntu1604:/usr/local/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop$ jps6034 DataNode6644 Jps6422 ResourceManager6536 NodeManager5915 NameNode6221 SecondaryNameNodehadoop@Ubuntu1604:/usr/local/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop$ ./sbin/mr-jobhistory-daemon.sh start historyserverstarting historyserver, logging to /usr/local/hadoop/logs/mapred-hadoop-historyserver-Ubuntu1604.outhadoop@Ubuntu1604:/usr/local/hadoop$ hadoop@Ubuntu1604:/usr/local/hadoop$ jps6816 JobHistoryServer6034 DataNode6917 Jps6422 ResourceManager6536 NodeManager5915 NameNode6221 SecondaryNameNodehadoop@Ubuntu1604:/usr/local/hadoop$ 访问Web页面
启用YARN之后,可以通过 Web 界面查看任务的运行情况：http://192.168.16.100:8088/cluster
关闭YARN和Hadoop
hadoop@Ubuntu1604:/usr/local/hadoop$ ./sbin/mr-jobhistory-daemon.sh stop historyserverhadoop@Ubuntu1604:/usr/local/hadoop$ ./sbin/stop-yarn.shhadoop@Ubuntu1604:/usr/local/hadoop$ ./sbin/stop-dfs.sh
