hadoop任务运行日志
来源:互联网 发布:string转数组 java 编辑:程序博客网 时间:2024/05/17 07:13
本环境中参数:
${yarn.app.mapreduce.am.staging-dir} : /user
NodeManager local log包括即container log
示例:
-bash-4.1
total 864K
-rw-r–r– 1 yarn yarn 0 Aug 26 10:08 stderr
-rw-r–r– 1 yarn yarn 0 Aug 26 10:08 stdout
-rw-r–r– 1 yarn yarn 864K Aug 26 15:19 syslog
JOB在运行的时候写在HDFS 的overview日志:
${yarn.app.mapreduce.am.staging-dir}/{username}/.staging/job_XXXXX_XXX/示例:-rw-r--r-- 10 dsp supergroup 18397766 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/job.jar-rw-r--r-- 10 dsp supergroup 22473 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/job.split-rw-r--r-- 3 dsp supergroup 864 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/job.splitmetainfo-rw-r--r-- 3 dsp supergroup 199357 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/job.xml-rw-r--r-- 3 dsp supergroup 333 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/job_1470221089121_737080.summary-rw-r--r-- 3 dsp supergroup 40623 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/job_1470221089121_737080_1.jhist-rw-r--r-- 3 dsp supergroup 229223 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/job_1470221089121_737080_1_conf.xmldrwx------ - dsp supergroup 0 2016-08-26 10:07 /user/dsp/.staging/job_1470221089121_737080/libjars
运行完成后,AM写入HDFS临时目录:
其中mapreduce.jobhistory.intermediate-done-dir默认值是${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate
示例:
-rwxrwx— 3 dsp hadoop 23629 2016-08-26 16:02 /user/history/done_intermediate/dsp/job_1470221089121_746317-1472198520907-dsp-from+NATIVE.HNATIVE_O…itle%2Cis_fund_dt_cust%28Stage-1472198529697-1-0-SUCCEEDED-root.dsp-1472198524249.jhist
-rwxrwx— 3 dsp hadoop 378 2016-08-26 16:02 /user/history/done_intermediate/dsp/job_1470221089121_746317.summary
-rwxrwx— 3 dsp hadoop 230331 2016-08-26 16:02 /user/history/done_intermediate/dsp/job_1470221089121_746317_conf.xml
日志收集线程进行日志整理写入的目录:
/user/history/done/2016/08/26/{xxx}
示例:
-rwxrwx— 3 dsp hadoop 23629 2016-08-26 16:02 /user/history/done/2016/08/26/000746/job_1470221089121_746317-1472198520907-dsp-from+NATIVE.HNATIVE_O…itle%2Cis_fund_dt_cust%28Stage-1472198529697-1-0-SUCCEEDED-root.dsp-1472198524249.jhist
-rwxrwx— 3 dsp hadoop 230331 2016-08-26 16:02 /user/history/done/2016/08/26/000746/job_1470221089121_746317_conf.xml
Hadoop 2.0中作业日志收集原理以及配置方法
- hadoop任务运行日志
- hadoop 命令行运行任务
- hadoop运行任务
- hadoop任务运行源代码深究
- hadoop提交任务后日志存放问题
- hadoop任务运行报错记录
- hadoop的jar命令运行任务方法
- 运行中hadoop任务修改/kill
- hadoop运行job任务出现异常
- Hadoop集群模式下运行Mapreduce任务
- hadoop中集群提交任务运行
- Hadoop系列三:运行job过程日志
- hadoop运行任务时出现网络链接异常
- Hadoop web项目使用Ajax监控MapReduce任务运行情况
- Hadoop 伪分布式模式 MapReduce 任务不能继续运行 解决方案
- Hadoop任务即将可以运行在Docker容器中
- hadoop本地任务运行过程报临时文件找不到问题排查
- hadoop节点运行的reduce和map任务数
- Android实现推送方式解决方案 - 长连接+心跳机制(MQTT协议)
- freemarker路径问题
- spring中实现Controller接口的映射
- C语言求解线性方程组AX=b
- 找不到org.apache.http等类
- hadoop任务运行日志
- Android 自定义ViewGroup之实现FlowLayout-标签流容器
- SQL显示查询结果的前100条
- spring中的jdbc连接用户增删改查
- JAVA中subString的用法
- OpenJudge 1750: 全排列
- 15 排序
- 写给失败的自己
- 个人js库