oozie配置sharelib,java.lang.NoClassDefFoundError: org/apache/pig/Main

来源:互联网 发布:淘宝改中差评 编辑:程序博客网 时间:2024/06/08 09:21

作为笔记:

sharelib的配置

在用oozie调用pig时,用到sharelib,当时是直接运行的,然后就报错了.

2013-07-03 20:44:48,504  WARN PigActionExecutor:542 - USER[root] GROUP[-] TOKEN[] APP[pig-wf] JOB[0000004-130703183035408-oozie-root-W] ACTION[0000004-130703183035408-oozie-root-W@pig-node] Launcher exception: org/apache/pig/Main
java.lang.NoClassDefFoundError: org/apache/pig/Main
at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:323)
at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:218)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:467)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: org.apache.pig.Main
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 17 more

最后在网上各种扒问题,发现是需要sharelib,action执行时,会对sharelib进行依赖,而sharelib要放到hdfs上。 
oozie提供一个sharelib:oozie-sharelib-**-incubating.tar.gz。解压即可。 

然后需要把sharelib放到hdfs上,hdfs的sharelib路径发现也是个问题。

这个需要在oozie安装目录下conf/oozie-default.xml中查找:

<property>
        <name>oozie.service.WorkflowAppService.system.libpath</name>
        <value>/user/${user.name}/share/lib</value>
        <description>
            System library path to use for workflow applications.
            This path is added to workflow application if their job properties sets
            the property 'oozie.use.system.libpath' to true.
        </description>
    </property> 

“/user/${user.name}/share/lib”就是我们需要放sharelib的路径。