(三)oozie官方案例map-reduce运行
来源:互联网 发布:人体芯片 知乎 编辑:程序博客网 时间:2024/06/07 17:14
oozie安装成功后,接下来运行一个官方的mapreduce案例,参考官网:
http://archive.cloudera.com/cdh5/cdh/5/oozie-4.0.0-cdh5.3.6/DG_Examples.html
1、解压oozie-example到oozie根目录
Expanding this file will create an examples/ directory in the local file system
examples下的子目录解析:
apps:所有自带的oozie案例,有mapreduce、hive、sqoop等等input-data:案例存放的测试数据src:案例源码
2、将exmples上传到HDFS的跟目录下
hdfs的根目录,即/uer/xusy
xusy—HDFS的用户名
命令:
$dfs -put examples examples
3、workflow解析
workflow.xml----->必须在hdfs上start...action...endjob.properties:属性信息 --->必须在本地-->需要根据实际配置lib:依赖包---->必须在本地
4、配置job.properties
nameNode=hdfs://xuxudede.com:9000jobTracker=xuxudede.com:8032queueName=defaultexamplesRoot=examplesoozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce/workflow.xmloutputDir=map-reduce
主要配置namenode、jobTracker;
其他的可以用默认!
nameNode:HDFS+namenode的端口号;
jobTracker:主机名+resourcemanager的端口号;
5、运行
$ oozie job -oozie http://xuxudede.com:11000/oozie -config examples/apps/map-reduce/job.properties -run
如果运行错误,请查看oozie.log日志
错误1:
2015-12-13 00:22:36,449 WARN ActionStartXCommand:544 - SERVER[xuxudede.com] USER[xusy] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-151213000534005-oozie-xus-W] ACTION[0000000-151213000534005-oozie-beif-W@mr-node] Error starting action [mr-node]. ErrorType [FAILED], ErrorCode [It should never happen], Message [File /user/xusy/share/lib does not exist]org.apache.oozie.action.ActionExecutorException: File /user/xusy/share/lib does not exist at org.apache.oozie.action.hadoop.JavaActionExecutor.addSystemShareLibForAction(JavaActionExecutor.java:631) at org.apache.oozie.action.hadoop.JavaActionExecutor.addAllShareLibs(JavaActionExecutor.java:709)
原因:
没有读取HDFS上的文件,读取的是本地文件系统上的
解决方法:
将hadoop的ect下的配置文件关联上oozie,配置如下:
<property> <name>oozie.service.HadoopAccessorService.hadoop.configurations</name> <value>*=/home/xusy/share/cdh5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop</value> <description> Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of the Hadoop service (JobTracker, HDFS). The wildcard '*' configuration is used when there is no exact match for an authority. The HADOOP_CONF_DIR contains the relevant Hadoop *-site.xml files. If the path is relative is looked within the Oozie configuration directory; though the path can be absolute (i.e. to point to Hadoop client conf/ directories in the local filesystem. </description> </property>
错误2:
2015-12-13 00:28:30,846 WARN ActionStartXCommand:544 - SERVER[xuxudede.com] USER[xusy] GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[0000000-151213002745581-oozie-beif-W] ACTION[0000000-151213002745581-oozie-xus-W@mr-node] Error starting action [mr-node]. ErrorType [NON_TRANSIENT], ErrorCode [JA002], Message [JA002: User: xusy is not allowed to impersonate xusy]org.apache.oozie.action.ActionExecutorException: JA002: User: xusy is not allowed to impersonate xusy
解决方法:
停止hadoop进程,重启linux主机,然后启动hadoop进程!
6.web上显示
正在运行:
url:xuxudede.com:11000
运行成功:
OK!mapreduce案例已经成功运行!
经验总结:不要怕遇到问题,要知道,每解决一个问题,你都会收获很多!
0 0
- (三)oozie官方案例map-reduce运行
- Hadoop Map/Reduce教程(官方文档)
- spark算子map reduce小案例
- filter()map() reduce()
- Map/Reduce运行时做了两次reduce的问题
- 快速获得Map/Reduce的运行结果
- 在cdh5-yarn上运行map-reduce
- hadoop学习6 运行map reduce出错
- 分布式计算(Map/Reduce)
- map/reduce
- map reduce
- Map/Reduce
- map reduce
- Map Reduce
- map reduce
- map-reduce
- Map-Reduce
- Map Reduce
- java jackson的使用
- Cookie和Session
- Opengl Uniform
- Django源码分析--1
- Design Pattern 之 迭代器模式
- (三)oozie官方案例map-reduce运行
- [java学习笔记]super的使用
- C++小题(二)
- PHP环境搭建
- Android中那些似懂非懂的常用知识点与好用的 Eclipse快捷键
- listview 常用的效果
- 如何简便计算两个空间向量之间的欧拉角
- Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected
- comet4j-java服务器推送实例