(三)oozie官方案例map-reduce运行

来源:互联网 发布:人体芯片 知乎 编辑:程序博客网 时间:2024/06/07 17:14

oozie安装成功后,接下来运行一个官方的mapreduce案例,参考官网:
http://archive.cloudera.com/cdh5/cdh/5/oozie-4.0.0-cdh5.3.6/DG_Examples.html

1、解压oozie-example到oozie根目录

Expanding this file will create an examples/ directory in the local file system
examples下的子目录解析:

apps:所有自带的oozie案例,有mapreduce、hive、sqoop等等input-data:案例存放的测试数据src:案例源码

2、将exmples上传到HDFS的跟目录下

hdfs的根目录,即/uer/xusy
xusy—HDFS的用户名
命令:

$dfs -put examples examples

3、workflow解析

workflow.xml----->必须在hdfs上start...action...endjob.properties:属性信息  --->必须在本地-->需要根据实际配置lib:依赖包---->必须在本地

4、配置job.properties

nameNode=hdfs://xuxudede.com:9000jobTracker=xuxudede.com:8032queueName=defaultexamplesRoot=examplesoozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce/workflow.xmloutputDir=map-reduce

主要配置namenode、jobTracker;
其他的可以用默认!
nameNode:HDFS+namenode的端口号;
jobTracker:主机名+resourcemanager的端口号;

5、运行

$ oozie job -oozie http://xuxudede.com:11000/oozie -config examples/apps/map-reduce/job.properties -run

这里写图片描述
如果运行错误,请查看oozie.log日志
错误1:

2015-12-13 00:22:36,449  WARN ActionStartXCommand:544 - SERVER[xuxudede.com] USER[xusy] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-151213000534005-oozie-xus-W] ACTION[0000000-151213000534005-oozie-beif-W@mr-node] Error starting action [mr-node]. ErrorType [FAILED], ErrorCode [It should never happen], Message [File /user/xusy/share/lib does not exist]org.apache.oozie.action.ActionExecutorException: File /user/xusy/share/lib does not exist        at org.apache.oozie.action.hadoop.JavaActionExecutor.addSystemShareLibForAction(JavaActionExecutor.java:631)        at org.apache.oozie.action.hadoop.JavaActionExecutor.addAllShareLibs(JavaActionExecutor.java:709)

原因:

没有读取HDFS上的文件,读取的是本地文件系统上的

解决方法:
将hadoop的ect下的配置文件关联上oozie,配置如下:

<property>        <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>        <value>*=/home/xusy/share/cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop</value>        <description>            Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of            the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is            used when there is no exact match for an authority. The HADOOP_CONF_DIR contains            the relevant Hadoop *-site.xml files. If the path is relative is looked within            the Oozie configuration directory; though the path can be absolute (i.e. to point            to Hadoop client conf/ directories in the local filesystem.        </description>    </property>

错误2:

2015-12-13 00:28:30,846  WARN ActionStartXCommand:544 - SERVER[xuxudede.com] USER[xusy] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-151213002745581-oozie-beif-W] ACTION[0000000-151213002745581-oozie-xus-W@mr-node] Error starting action [mr-node]. ErrorType [NON_TRANSIENT], ErrorCode [JA002], Message [JA002: User: xusy is not allowed to impersonate xusy]org.apache.oozie.action.ActionExecutorException: JA002: User: xusy is not allowed to impersonate xusy

解决方法:
停止hadoop进程,重启linux主机,然后启动hadoop进程!

6.web上显示

正在运行:
url:xuxudede.com:11000
这里写图片描述
运行成功:
这里写图片描述

OK!mapreduce案例已经成功运行!
经验总结:不要怕遇到问题,要知道,每解决一个问题,你都会收获很多!

0 0
原创粉丝点击