WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
来源:互联网 发布:筷子兄弟 知乎 编辑:程序博客网 时间:2024/06/07 14:58
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
搜索的结果大部分说是操作系统与hadoop的位数不一致造成的,然而事实并非如此。
本文的操作系统为centos 64位,而libhadoop.so也是64位的
$ file ../lib/native/libhadoop.so.1.0.0
../lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
后来甚至在机器上耗费数天从编译hadoop到编译spark,最后依然会提示上述warning,那么显然问题并不是位数冲突造成的了。
由于上述warning并不能提供更多有效的信息,需要增加debug信息:
export HADOOP_ROOT_LOGGER=DEBUG,console
再次执行./stop-dfs.sh,这次debug到一些有用的信息:
16/07/05 15:33:54 DEBUG util.Shell: setsid exited with exit code 016/07/05 15:33:54 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[Rate of successful kerberos logins and latency (milliseconds)], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops)16/07/05 15:33:54 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[Rate of failed kerberos logins and latency (milliseconds)], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops)16/07/05 15:33:54 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[GetGroups], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops)16/07/05 15:33:54 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics16/07/05 15:33:55 DEBUG security.Groups: Creating new Groups object16/07/05 15:33:55 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...<span style="color:#FF0000;">16/07/05 15:33:55 DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path16/07/05 15:33:55 DEBUG util.NativeCodeLoader: java.library.path=/usr/local/sinasrv2/hadoop/hadoop-2.6.4/lib16/07/05 15:33:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable</span>16/07/05 15:33:55 DEBUG util.PerformanceAdvisory: Falling back to shell based16/07/05 15:33:55 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping16/07/05 15:33:55 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=500016/07/05 15:33:55 DEBUG security.UserGroupInformation: hadoop login16/07/05 15:33:55 DEBUG security.UserGroupInformation: hadoop login commit16/07/05 15:33:55 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: hadoop16/07/05 15:33:55 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: hadoop" with name hadoop16/07/05 15:33:55 DEBUG security.UserGroupInformation: User entry: "hadoop"16/07/05 15:33:55 DEBUG security.UserGroupInformation: UGI loginUser:hadoop (auth:SIMPLE)16/07/05 15:33:55 DEBUG security.UserGroupInformation: PrivilegedAction as:hadoop (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:314)
可以看到是上述标红位置出现了问题。也就是从java.library.path处没有找到libhadoop.so,我们发现,libhadoop.so是存放在/usr/local/sinasrv2/hadoop/hadoop-2.6.4/lib/native下的
在hadoop包中搜索发现,java.library.path被定义为$JAVA_LIBRARY_PATH
$ grep -R 'java.library.path' *bin/yarn: YARN_OPTS="$YARN_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"bin/yarn.cmd: set YARN_OPTS=%YARN_OPTS% -Djava.library.path=%JAVA_LIBRARY_PATH%etc/hadoop/yarn-env.sh: YARN_OPTS="$YARN_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"etc/hadoop/yarn-env.cmd: set YARN_OPTS=%YARN_OPTS% -Djava.library.path=%JAVA_LIBRARY_PATH%libexec/hadoop-config.sh:# setup 'java.library.path' for native-hadoop code if necessarylibexec/hadoop-config.sh: HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"libexec/hadoop-config.cmd:@rem setup 'java.library.path' for native hadoop code if necessarylibexec/hadoop-config.cmd: set HADOOP_OPTS=%HADOOP_OPTS% -Djava.library.path=%JAVA_LIBRARY_PATH%share/doc/hadoop/common/CHANGES.txt: HADOOP-8756. Fix SEGV when libsnappy is in java.library.path butshare/doc/hadoop/common/CHANGES.txt: HADOOP-1660. Add the cwd of the map/reduce task to the java.library.pathshare/doc/hadoop/common/CHANGES.txt: 50. HADOOP-1493. Permit specification of "java.library.path" systemshare/doc/hadoop/common/CHANGES.txt:10. HADOOP-873. Pass java.library.path correctly to child processes.share/doc/hadoop/common/CHANGES.txt:42. HADOOP-838. Fix tasktracker to pass java.library.path toshare/doc/hadoop/mapreduce/CHANGES.txt: MAPREDUCE-4458. Warn if java.library.path is used for AM or Taskshare/doc/hadoop/mapreduce/CHANGES.txt: MAPREDUCE-4072. User set java.library.path seems to overwrite defaultshare/doc/hadoop/mapreduce/CHANGES.txt: MAPREDUCE-3259. Added java.library.path of NodeManager to
但是此时发现环境变量$JAVA_LIBRARY_PATH并未定义,那么问题到这里终于算是找到了,解决方法也很简单了,vim /etc/profile,添加$JAVA_LIBRARY_PATH环境变量
export JAVA_LIBRARY_PATH='/usr/local/sinasrv2/hadoop/hadoop-2.6.4/lib/native'
source /etc/profile 使之生效
再执行,则不会出现上述warning信息
2、在执行spark-shell的时候出现上述warning信息
同样的思路,修改spark的日志级别为warning,vim $SPARK_HOME/conf/log4j.properties
log4j.rootCategory=WARNING, console
得到如下错误信息:
16/07/06 12:15:41 DEBUG Groups: Creating new Groups object16/07/06 12:15:41 DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library...16/07/06 12:15:41 DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path16/07/06 12:15:41 DEBUG NativeCodeLoader: java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib16/07/06 12:15:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable16/07/06 12:15:41 DEBUG PerformanceAdvisory: Falling back to shell based
又是java.library.path的问题,Orz…
但是在grep的时候,没有发现java.library.path的出处,也不知道其值是如何设定的。问题虽然找到了,但是解决方法一筹莫展。只要求助万能的度娘了…
这篇帖子也是同样的问题,http://bbs.csdn.net/topics/390978577,采用7楼的解决方法:
在spark的conf目录下,修改spark-env.sh文件加入LD_LIBRARY_PATH环境变量,值为hadoop的native库的路径
果然凑效!
vim $SPARK_HOME/conf/spark-env.sh
export SPARK_MASTER_IP=hadoop1 export SPARK_MASTER_PORT=7077 export SPARK_WORKER_CORES=1 export SPARK_WORKER_INSTANCES=1 export SPARK_WORKER_MEMORY=512M export LD_LIBRARY_PATH=$JAVA_LIBRARY_PATH
- hadoop WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using b
- Hadoop WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using b
- Hadoop之—— WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
- Hadoop 2.7.2启动WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
- 【解决办法】hadoop运行warn “util.NativeCodeLoader: Unable to load native-hadoop library for your platform”
- Hadoop之—— WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
- Hadoop之—— WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
- Hadoop报错---WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
- hadoop WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... usin
- Hadoop之—— WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
- Hadoop之—— WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
- Hadoop伪分布式WARN util.NativeCodeLoader:Unable to load native-hadoop library for your platform...usi
- Hadoop问题: WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
- WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-
- 解决WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform问题
- 错误:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using built
- WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...的解决方案
- WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-
- Android5.0以上版本查看应用使用情况
- 决策树学习笔记(二)
- BZOJ 3037 创世纪
- zookeeper原理
- swift 将json字符串 转换成 字典
- WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
- python import cv2报错 segmentation fault:11
- 46种常见的浏览器兼容性问题大汇总
- 新手写openmp程序,结果不能运行,求教高手指点
- Remove Element
- jQuery选择器 --- File
- HBase 常用Shell命令
- Problem H
- 利用JAVA反射机制解析XML数据到实体