【hadoop】 1005-yarn测试

1、启动yarn

[hadoop@cloud01 hadoop-2.4.1]$ sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/app/hadoop-2.4.1/logs/yarn-hadoop-resourcemanager-cloud01.out
hadoop@localhost's password:
localhost: starting nodemanager, logging to /home/hadoop/app/hadoop-2.4.1/logs/yarn-hadoop-nodemanager-cloud01.out

2、查看启动的进程

[hadoop@cloud01 hadoop-2.4.1]$ jps
6113 SecondaryNameNode
6659 NodeManager
5970 DataNode
6760 Jps
6355 ResourceManager
5855 NameNode

ResourceManager：整个yarn的资源调度和管理

NodeManager：本节点资源调度和管理

3、YARN管理界面

 http://192.168.2.31:8088

4、通过程序验证

4.1 创建words.txt

[hadoop@cloud01 ~]$ more words.txt
hello tom
hello jerry
hello tom
hello world

4.2 上传words.txt到hdfs上

[hadoop@cloud01 ~]$ hdfs dfs -put words.txt hdfs://cloud01:9000/

4.3 使用hadoop内置程序，提交给yarn

[hadoop@cloud01 mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.4.1.jar wordcount  /words.txt /output

15/02/01 15:28:20 INFO client.RMProxy: Connecting to ResourceManager at cloud01/192.168.2.31:8032
15/02/01 15:28:21 INFO input.FileInputFormat: Total input paths to process : 1
15/02/01 15:28:22 INFO mapreduce.JobSubmitter: number of splits:1
15/02/01 15:28:23 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1422831663687_0002
15/02/01 15:28:23 INFO impl.YarnClientImpl: Submitted application application_1422831663687_0002
15/02/01 15:28:23 INFO mapreduce.Job: The url to track the job: http://cloud01:8088/proxy/application_1422831663687_0002/
15/02/01 15:28:23 INFO mapreduce.Job: Running job: job_1422831663687_0002
15/02/01 15:28:45 INFO mapreduce.Job: Job job_1422831663687_0002 running in uber mode : false
15/02/01 15:28:45 INFO mapreduce.Job:  map 0% reduce 0%
15/02/01 15:29:13 INFO mapreduce.Job:  map 100% reduce 0%
15/02/01 15:29:30 INFO mapreduce.Job:  map 100% reduce 100%
15/02/01 15:29:33 INFO mapreduce.Job: Job job_1422831663687_0002 completed successfully
15/02/01 15:29:34 INFO mapreduce.Job: Counters: 49
     File System Counters
          FILE: Number of bytes read=52
          FILE: Number of bytes written=185933
          FILE: Number of read operations=0
          FILE: Number of large read operations=0
          FILE: Number of write operations=0
          HDFS: Number of bytes read=138
          HDFS: Number of bytes written=30
          HDFS: Number of read operations=6
          HDFS: Number of large read operations=0
          HDFS: Number of write operations=2
     Job Counters
          Launched map tasks=1
          Launched reduce tasks=1
          Data-local map tasks=1
          Total time spent by all maps in occupied slots (ms)=26769
          Total time spent by all reduces in occupied slots (ms)=11213
          Total time spent by all map tasks (ms)=26769
          Total time spent by all reduce tasks (ms)=11213
          Total vcore-seconds taken by all map tasks=26769
          Total vcore-seconds taken by all reduce tasks=11213
          Total megabyte-seconds taken by all map tasks=27411456
          Total megabyte-seconds taken by all reduce tasks=11482112
     Map-Reduce Framework
          Map input records=4
          Map output records=8
          Map output bytes=76
          Map output materialized bytes=52
          Input split bytes=94
          Combine input records=8
          Combine output records=4
          Reduce input groups=4
          Reduce shuffle bytes=52
          Reduce input records=4
          Reduce output records=4
          Spilled Records=8
          Shuffled Maps =1
          Failed Shuffles=0
          Merged Map outputs=1
          GC time elapsed (ms)=308
          CPU time spent (ms)=1570
          Physical memory (bytes) snapshot=203378688
          Virtual memory (bytes) snapshot=725680128
          Total committed heap usage (bytes)=126627840
     Shuffle Errors
          BAD_ID=0
          CONNECTION=0
          IO_ERROR=0
          WRONG_LENGTH=0
          WRONG_MAP=0
          WRONG_REDUCE=0
     File Input Format Counters
          Bytes Read=44
     File Output Format Counters
          Bytes Written=30

4.4  通过hadoop jar 提交yarn的过程，进程变化

[hadoop@cloud01 ~]$ jps
6113 SecondaryNameNode
7689 Jps
6659 NodeManager
5970 DataNode
7541 RunJar
7679 MRAppMaster
6355 ResourceManager
5855 NameNode

        YarnChild

RunJar： hadoop jar 时，作为一个客户端提交给yarn

ResourceManager：提交给YARN资源管理器

NodeManager： 每个执行节点管理器

MRAppMaster： 启动一个MapReduce通过该进程完成

YarnChild： 执行MapReduce的进程

4.5 验证查看结果

一、通过YARN管理界面

二、通过命令行方式查看

[hadoop@cloud01 ~]$ more words.txt
hello tom
hello jerry
hello tom
hello world

[hadoop@cloud01 ~]$ hdfs dfs -text /output/part-r-00000
hello     4
jerry     1
tom     2
world     1