oozie框架案例之shellAction
来源:互联网 发布:下载淘宝助理免费 编辑:程序博客网 时间:2024/06/05 15:41
oozie配置Shell脚本调度
MapReduce
Hive
1、拷贝模版
$ cp -r examples/apps/shell/ oozie-apps
nameNode=hdfs://[hostname]:8020
jobTracker=[hostname]:8032
queueName=default
examplesRoot=oozie-apps
oozie.wf.application.path=${nameNode}/user/admin/${examplesRoot}/shell
EXEC=mem.sh
<start to="shell-node"/>
<action name="shell-node">
<shell xmlns="uri:oozie:shell-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>${EXEC}</exec>
<file>${nameNode}/user/user01/${examplesRoot}/shell/${EXEC}#${EXEC}</file>
</shell>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
4、创建测试脚本
#!/bin/sh
/bin/date -R >> /opt/modules/cdh/oozie-4.0.0-cdh5.3.6/oozie-apps/shell/
1.log
/usr/bin/free -m >> /opt/modules/cdh/oozie-4.0.0-cdh5.3.6/oozie-apps/sh
ell/1.log
/bin/df -l >> /opt/modules/cdh/oozie-4.0.0-cdh5.3.6/oozie-apps/shell/1.
log
echo ------------------- >> /opt/modules/cdh/oozie-4.0.0-cdh5.3.6/oozie
-apps/shell/1.log
5.上传到HDFS对应目录
$ bin/hdfs dfs -put /opt/modules/oozie-4.0.0-cdh5.3.6/oozie-apps/shell /user/user01/oozie-apps/
6、执行:
$ bin/oozie job -oozie http://[hostname]:11000/oozie -config oozie-apps/shell/job.properties -run
$ bin/oozie job -oozie http://[hostname]:11000/oozie -config oozie-apps/shell/job.properties -run
以上是配置oozie的手动提交 --workflow
MapReduce
Hive
1、拷贝模版
$ cp -r examples/apps/shell/ oozie-apps
2、=========修改job.properties============
The corresponding job properties file used to submit Oozie job could be as follows:
oozie.wf.application.path=hdfs://localhost:8020/user/kamrul/workflows/script#Execute is expected to be in the Workflow directory.#Shell Script to runEXEC=script.sh#CPP executable. Executable should be binary compatible to the compute node OS.#EXEC=hello#Perl script#EXEC=script.pljobTracker=localhost:8021nameNode=hdfs://localhost:8020queueName=default
nameNode=hdfs://[hostname]:8020
jobTracker=[hostname]:8032
queueName=default
examplesRoot=oozie-apps
oozie.wf.application.path=${nameNode}/user/admin/${examplesRoot}/shell
EXEC=mem.sh
3、修改==============workflow.xml=============
How to run any shell script or perl script or CPP executable
<workflow-app xmlns='uri:oozie:workflow:0.3' name='shell-wf'> <start to='shell1' /> <action name='shell1'> <shell xmlns="uri:oozie:shell-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <exec>${EXEC}</exec> <argument>A</argument> <argument>B</argument> <file>${EXEC}#${EXEC}</file> <!--Copy the executable to compute node's current working directory --> </shell> <ok to="end" /> <error to="fail" /> </action> <kill name="fail"> <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name='end' /></workflow-app>
<start to="shell-node"/>
<action name="shell-node">
<shell xmlns="uri:oozie:shell-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>${EXEC}</exec>
<file>${nameNode}/user/user01/${examplesRoot}/shell/${EXEC}#${EXEC}</file>
</shell>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
4、创建测试脚本
#!/bin/sh
/bin/date -R >> /opt/modules/cdh/oozie-4.0.0-cdh5.3.6/oozie-apps/shell/
1.log
/usr/bin/free -m >> /opt/modules/cdh/oozie-4.0.0-cdh5.3.6/oozie-apps/sh
ell/1.log
/bin/df -l >> /opt/modules/cdh/oozie-4.0.0-cdh5.3.6/oozie-apps/shell/1.
log
echo ------------------- >> /opt/modules/cdh/oozie-4.0.0-cdh5.3.6/oozie
-apps/shell/1.log
5.上传到HDFS对应目录
$ bin/hdfs dfs -put /opt/modules/oozie-4.0.0-cdh5.3.6/oozie-apps/shell /user/user01/oozie-apps/
6、执行:
$ bin/oozie job -oozie http://[hostname]:11000/oozie -config oozie-apps/shell/job.properties -run
$ bin/oozie job -oozie http://[hostname]:11000/oozie -config oozie-apps/shell/job.properties -run
以上是配置oozie的手动提交 --workflow
阅读全文
0 0
- oozie框架案例之shellAction
- oozie框架案例之多Action执行
- 大数据协作框架之Oozie篇
- 大数据协作框架之Oozie
- 工作流调度框架Oozie
- oozie workflow.xml 综合案例
- 驭象者之Apache Oozie
- hadoop之oozie配置
- 大数据作协框架Oozie
- Oozie
- Oozie
- OOZIE
- oozie
- oozie
- oozie
- oozie
- Oozie
- Oozie工作流之理论篇
- 智能指针(一)之引入
- mysql的单引号和双引号的区别??(基本没有)
- 需求处理的妥协和更好的设计过程模型
- hdu 1285 确定比赛名次
- 多线程问题(生产者和消费者)
- oozie框架案例之shellAction
- 背包四讲及其优化与证明
- qwt绘制,以及画面缩放、平移操作
- 蓄水池抽样算法 (Reservoir Sampling Algorithm)
- 小白学tkinter(Listbox组件 多选组件)
- 鹧鸪天·彩袖殷勤捧玉钟
- 大暴搜 [NOIP2009]靶形数独
- C语言控制鼠标自动画图
- java进程负载高情况排查