Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
来源:互联网 发布:电梯维修需要编程吗 编辑:程序博客网 时间:2024/05/29 14:29
转:http://stackoverflow.com/questions/19323611/simple-oozie-shell-action-from-rest-api-fails
Simple oozie shell action from REST API fails
I'm using oozie's REST API to invoke shell script actions. Any script that includes accessing or using HDFS fails.
I test with this simple script
#!/bin/bash -ehadoop fs -touchz aaa
When i run the Oozie workflow that invokes this script from command line -- as "root" -- it works fine. When i invoke it from the REST API, it fails with Error Code JA018 and Error Message:
Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
I am certain that my oozie assets are set up properly (i.e., job.properties and workflow.xml) since if i swap out "touch /tmp/aaa" (i.e., a linux touch command) the workflow runs to successful completion.
First, i can't find clear documentation of what JA018 actually is.
Second (and mainly), I think the cause of my error may be permissions. When the script works (i.e., using linux touch command -- not HDFS touchz), I'm passing the job property user.name as "root" to the Oozie REST server. The Oozie REST server (running as "oozie" per Cloudera configuration) does not run the job as the "user.name" passed to it (else presumably it'd be running the workflow as "root" and it'd work). Is there some way i need to configure oozie ID to be able to run as the user.name I pass to it? Should I use an ID other than root? (I use root precisely because it has all permissions.)
Seems this is a problem others would have, but i've found next to nothing on the web. There is probably some standard way of doing this that i am simply missing.
I am using Cloudera 4.1.3.
thanks for any thoughts / suggestions.
1 Answer
My approach and assumptions were wrong -- thus the problem. Turns out that Oozie doesn't support this sort of thing because the shell script is being run as a different user to the one that Oozie is doing it's stuff internally in HDFS as. In other words, running the shell script loses the context in which Oozie does operations within HDFS. So, having Oozie invoke a shell script and expecting it to be able to do things in HDFS is the wrong approach.
Oozie is kinda limited in what it can do explicitly in HDFS, but it appears that is the way to invoke HDFS actions within Oozie.
- Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
- Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [10]
- error:org.apache.oozie.action.ActionExecutorException:Permissiondenied
- oozie配置sharelib,java.lang.NoClassDefFoundError: org/apache/pig/Main
- Could not find the main class: org.apache.spark.launcher.Main. Program will exit.
- Oozie Error: E0902 : E0902: Exception occured: [org.apache.hadoop.ipc.RemoteException: User: oozie i
- Apache Oozie Workflow Scheduler for Hadoop
- Apache Oozie - the workflow scheduler for hadoop
- Oozie
- Oozie
- OOZIE
- oozie
- oozie
- oozie
- oozie
- Oozie
- Hadoop: Hadoop oozie main sub workflow.xml configuration
- oozie hive action注意事项
- hive0.12.0配置与安装详解
- js数组与字符串的相互转换方法
- CrapApi接口管理系统菜单设置
- interpret-only模式改为speed模式的测试时间对比
- 断点续传&网络编程day2(66期第五天)
- Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
- 编码-京东实习笔试编程题-糖果问题-动态规划
- Oracle-10-对表中数据进行增删改查
- 关联(objc_setAssociatedObject、objc_getAssociatedObject、objc_removeAssociatedObjects)
- 238. Product of Array Except Self
- LeetCode(54)-Longest Common Prefix
- ubuntu使用下载的deb文件制作本地源
- java获取文件名的三种方法
- 百钱买百鸡问题