Zeppelin 0.6.2 使用spark2.x 的一些错误处理

来源：互联网发布：matlab 矩阵方程编辑：程序博客网时间：2024/06/07 23:38

zeppelin 从 0.6.1 开始支持 spark2.x ，从0.6.2编译时引入了hadoop-common包用于权限认证，所以会存在一些包冲突导致异常的问题。

编译错误

zeppelin-web编译错误

Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.25:grunt (grunt test) on project zeppelin-web: Failed to run task: 'grunt test --no-color' failed. (error code 3) -> [Help 1]

修改zepelin-web下的pom文件，删除以下内容

<execution><id>grunt test</id><goals>    <goal>grunt</goal></goals><phase>test</phase><configuration>    <arguments>test</arguments></configuration></execution>

运行时报错

启动spark interpreter报错，hadoop包冲突

错误提示1：

java.lang.NoSuchMethodError: org.apache.hadoop.fs.FSOutputSummer.<init>(Ljava/util/zip/Checksum;II)V

错误提示2：

java.lang.NoSuchMethodError: org.apache.hadoop.tracing.SpanReceiverHost.get(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/String;)Lorg/apache/hadoop/tracing/SpanReceiverHost;at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:634) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)

自带了hadoop2.6的common包与spark的hadoop包冲突。使用spark的包替换即可。

cd zeppelin/librm hadoop-annotations-2.6.0.jarrm hadoop-auth-2.6.0.jarrm hadoop-common-2.6.0.jarcp spark/jars/hadoop-annotations-2.4.1.jar ./cp spark/jars/hadoop-auth-2.4.1.jar ./cp spark/jars/hadoop-common-2.4.1.jar ./

运行spark任务报错，jackson包冲突

报错提示:

com.fasterxml.jackson.databind.JsonMappingException: Jackson version is too old 2.5.3

解决方法:

cd zeppelin/librm jackson-annotations-2.5.0.jarrm jackson-core-2.5.3.jarrm jackson-databind-2.5.3.jarcp spark/jars/jackson-databind-2.6.5.jar ./cp spark/jars/jackson-core-2.6.5.jar ./cp spark/jars/jackson-annotations-2.6.5.jar ./

0 0