spark on yarn报错ERROR GPLNativeCodeLoader: Could not load native gpl library

来源：互联网发布：如何在淘宝上买到好茶编辑：程序博客网时间：2024/05/13 13:36

启动spark-sql
spark-sql --master yarn-client --executor-memory 1G

报错：
16/03/30 14:45:32 WARN BlockReaderLocal: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.

使用命令查看databases
spark-sql> show databases;

16/03/30 14:46:01 ERROR GPLNativeCodeLoader: Could not load native gpl library
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
        at java.lang.Runtime.loadLibrary0(Runtime.java:849)
        at java.lang.System.loadLibrary(System.java:1088)
        at com.hadoop.compression.lzo.GPLNativeCodeLoader.(GPLNativeCodeLoader.java:32)
        at com.hadoop.compression.lzo.LzoCodec.(LzoCodec.java:71)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:270)
        at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1834)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1799)
        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
        at org.apache.hadoop.io.compress.CompressionCodecFactory.(CompressionCodecFactory.java:175)
        at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

16/03/30 14:46:01 ERROR LzoCodec: Cannot load native-lzo without native-hadoop

解决方法：
spark-env.sh中增加
export SPARK_SUBMIT_LIBRARY_PATH=/usr/lib/hadoop/lib/native/:$SPARK_SUBMIT_LIBRARY_PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/hadoop/lib/native/
export SPARK_LIBRARY_PATH=/usr/lib/hadoop/lib/native/:$SPARK_LIBRARY_PATH
export SPARK_CLASSPATH=/usr/local/spark/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar:$CLASSPATH

后来尝试了下，保留以下两个即可：
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/hadoop/lib/native/
export SPARK_CLASSPATH=/usr/local/spark/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar:$CLASSPATH

参考：
http://find.searchhub.org/document/a128707a98fe4ec6
http://lotso.blog.51cto.com/3681673/1441737
http://guoyunsky.iteye.com/blog/1237327

0 0