hadoop任务卡死

hadoop 运行mapreduce的时候会卡死在 mapreduce.Job:Running job: job_1477030467429_0002  位置不动

思路一：分析：mapreduce卡死不动，原可能是  resourcemanager 或者 nodemanager 配置出错

检查yarn-site.xml（yarn.resourcemanager.hostname：配置了resourcemanager 的位置） 或者 slaves （配置了nodemanager 位置）配置文件

可能出现的错误：

  yarn.resourcemanager.hostname ：配置出错；

  slaves  没有加入namenode节点；

  hosts 配置出错

 

思路二：运行内存不足的问题 ：

现象：Memory Total  为 0；

 日志：

2016-10-29 10:28:30,433 WARN org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection: Directory /opt/cdh/hadoop-2.5.0-cdh5.3.6/data/tmp/nm-local-dir error, used space above threshold of 90.0%, removing from list of valid directories

2016-10-29 10:28:30,433 WARN org.apache.hadoop.yarn.server.nodemanager.DirectoryCollection: Directory /opt/cdh/hadoop-2.5.0-cdh5.3.6/logs/userlogs error, used space above threshold of 90.0%, removing from list of valid directories

2016-10-29 10:28:30,433 INFO org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Disk(s) failed: 1/1 local-dirs are bad: /opt/cdh/hadoop-2.5.0-cdh5.3.6/data/tmp/nm-local-dir; 1/1 log-dirs are bad: /opt/cdh/hadoop-2.5.0-cdh5.3.6/logs/userlogs

2016-10-29 10:28:30,435 ERROR org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Most of the disks failed. 1/1 local-dirs are bad: /opt/cdh/hadoop-2.5.0-cdh5.3.6/data/tmp/nm-local-dir; 1/1 log-dirs are bad: /opt/cdh/hadoop-2.5.0-cdh5.3.6/logs/userlogs

2016-10-29 10:28:31,204 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1477706304072_0002_01_000001

2016-10-29 10:28:31,558 WARN org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: Unexpected: procfs stat file is not in the expected format for process with pid 3202

2016-10-29 10:28:31,581 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 11809 for container-id container_1477706304072_0002_01_000001: 33.0 MB of 2 GB physical memory used; 1.6 GB of 4.2 GB virtual memory used

2016-10-29 10:28:31,910 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1477706304072_0002_01_000001 transitioned from RUNNING to KILLING

2016-10-29 10:28:31,910 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1477706304072_0002_01_000001

2016-10-29 10:28:31,954 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1477706304072_0002_01_000001 is : 143

2016-10-29 10:28:31,989 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1477706304072_0002_01_000001 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL

2016-10-29 10:28:31,989 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /opt/cdh/hadoop-2.5.0-cdh5.3.6/data/tmp/nm-local-dir/usercache/kequan/appcache/application_1477706304072_0002/container_1477706304072_0002_01_000001

2016-10-29 10:28:31,991 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=kequan       OPERATION=Container Finished - Killed   TARGET=ContainerImpl    RESULT=SUCCESS      APPID=application_1477706304072_0002    CONTAINERID=container_1477706304072_0002_01_000001

2016-10-29 10:28:31,991 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1477706304072_0002_01_000001 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE

2016-10-29 10:28:31,992 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1477706304072_0002_01_000001 from application application_1477706304072_0002

2016-10-29 10:28:31,992 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_1477706304072_0002_01_000001 for log-aggregation

2016-10-29 10:28:31,992 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1477706304072_0002

2016-10-29 10:28:32,915 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1477706304072_0002_01_000001]

2016-10-29 10:28:34,582 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1477706304072_0002_01_000001

 

分析： used space above thresholdof 90.0%  磁盘空间超过90% ，MR运行很占用磁盘空间，磁盘空间不够用的时候，nodemanager被强行杀死；

  方法一：设置磁盘最高利用率为 95 ；在yarn-site.xml目录里加入下面的配置（治标不治本，MR运行时候，磁盘使用空间还有可能超过 95%）

<property>

       <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>

        <value>95.0</value>

    </property>

 

方法二： 删除磁盘里面不用的空间

                在命令行执行  df -h 查看磁盘用了多少空间

[root@hadoop-senior01 modules]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda2              18G   14G  3.4G  80% /

tmpfs                 1.9G  372K  1.9G   1% /dev/shm

/dev/sda1             291M   37M  240M  14% /boot

     /dev/sda2   磁盘 就是指 系统所有内存；删除系统里面不用的文件或者软件即可

 

 ERRORorg.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService: Most of thedisks failed. 1/1 local-dirs are bad:/opt/cdh/hadoop-2.5.0-cdh5.3.6/data/tmp/nm-local-dir; 1/1 log-dirs are bad:/opt/cdh/hadoop-2.5.0-cdh5.3.6/logs/userlogs

       参考： http://www.cnblogs.com/tnsay/p/5917459.html

方法三： 如果是虚拟机那么久扩容；真机加磁盘

    虚拟机扩容： http://www.2cto.com/os/201405/301879.html