Windows 平台运行spark-shell 报"java.lang.NullPointerException, not found: value sqlContext" error 解决办法
来源:互联网 发布:js文本框提示信息 编辑:程序博客网 时间:2024/06/06 09:40
错误背景描述
在spark 官网下载 spark的安装版:spark-1.6.3-bin-hadoop2.6.tgz,然后解压,配好环境变量,在window下CMD下运行spark-shell,启动spark失败(本地模式)。然后搜索了一下,发现中文基本找不到什么参考资料,不过在stackoverflow上找到了解决办法,遂记录下,供国内的同学参考下
错误信息
ava.lang.NullPointerException (not found: value sqlContext)java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:194)at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)at org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:462)at org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:461)at org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330)at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:422)at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028)at $iwC$$iwC.<init>(<console>:15)at $iwC.<init>(<console>:24)at <init>(<console>:26)at .<init>(<console>:30)at .<clinit>(<console>)at .<init>(<console>:7)at .<clinit>(<console>)at $print(<console>)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:497)at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:132)at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991)at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)at org.apache.spark.repl.Main$.main(Main.scala:31)at org.apache.spark.repl.Main.main(Main.scala)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:497)at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused by: java.lang.NullPointerExceptionat java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)at org.apache.hadoop.util.Shell.run(Shell.java:455)at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)at org.apache.hadoop.util.Shell.execCommand(Shell.java:808)at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097)at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:582)at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:557)at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:599)at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)… 62 more<console>:16: error: not found: value sqlContextimport sqlContext.implicits._^<console>:16: error: not found: value sqlContextimport sqlContext.sql
错误解决办法
出现这种错误通常是因为缺少 winutils.exe文件,而这个文件是Spark用来初始化Hive的上下文,而这又取决于Hadoop,这就需要Windows上的本机库正常工作。不幸的是,即使您在本地模式下使用Spark而不直接使用任何HDFS功能,也会发生这种情况。所以为了解决这个问题,你需要按如下步骤做。
下载winutils.exe,注意32位要下载32位的,64位要下载64位的
download the 64-bit winutils.exe 32位的自己去github上找找都能找到的在你的本地建立hadoop/bin 文件夹,然后把winutils.exe放进去。比如我的是C:\hadoop\bin,然后去建立一个系统环境变量HADOOP_HOME,把你刚才的新建的文件夹的路径去掉bin ,例如:文件夹路径是C:\hadoop\bin那么HADOOP_HOME就是C:\hadoop,文件夹路径是C:\spark\hadoop\bin那么
HADOOP_HOME就是C:\spark\hadoop,建好系统变量以后去CMD下运行
echo %HADOOP_HOME%,检查是否正确。
3.检查你的C盘是否有C:\tmp\hive文件夹,没有新建之。一般都是有的,然后右键选择管理员获取所有权,最后在CMD下运行一下命令就可以了
这个命令的意思是让winutils.exe文件对 \tmp\hive有读写权限。
%HADOOP_HOME%\bin\winutils.exe chmod 777 \tmp\hive
4.重新运行spark-shell 会出现一下画面就表示启动成功了
如果不放心,在浏览器中运行http://localhost:4040出现一下画面也表示启动成功。
参考英文资料:
- stackoverflow
- blogs.msdn.microsoft.com
- Windows 平台运行spark-shell 报"java.lang.NullPointerException, not found: value sqlContext" error 解决办法
- 在spark中使用Hive报错error: not found: value sqlContext
- spark-shell 启动报错, error: not found: value spark(低级已解决)
- 关于启动spark-shell报错:<console>:14: error: not found: value spark
- java.lang.NullPointerException: Module 'null' not found
- java.lang.NullPointerException: Module 'null' not found .
- java.lang.NullPointerException: Module 'null' not found.
- win7 spark运行本地程序文件出错 error:avaSparkContext. : java.lang.NullPointerException
- myeclipse报An error has occurred,See error log for more details. java.lang.NullPointerException错误解决办法
- getHibernateTemplate()中的方法报java.lang.NullPointerException的解决办法
- java.lang.NullPointerException(万普平台SDK出错解决办法)
- Windows平台的 PHP 报错 Fatal error: Class COM not found in 的解决方法
- Android JNI报错:java.lang.UnsatisfiedLinkError: Native method not found ..........解决办法
- loadjava报错Java: Not Found Error
- Stacktraces 报 java.lang.NullPointerException
- 报java.lang.NullPointerException原因
- Unknown error: java.lang.NullPointerException.
- Unknown error: java.lang.NullPointerException.
- PreparedStatement与Statement理解
- android TextView遇到的问题
- java学习之路之基本语法-程序流程控制
- java if语句练习
- 《Spark商业案例与性能调优实战100课》第1课:商业案例之通过RDD实现分析大数据电影点评系统中电影的用户行为信息
- Windows 平台运行spark-shell 报"java.lang.NullPointerException, not found: value sqlContext" error 解决办法
- [1]《微交互》读感 —— 往小处想,改变世界!
- WindowsPowerShell 出现无法识别cmdlet、函数、脚本等错误
- Java注解处理器 - 五分钟快速入门
- opencv学习笔记(十四)图像金字塔
- HTTP返回码
- MySQL binlog 组提交与 XA(两阶段提交)
- Windows平台下,将mdb文件快速向MySQL中导入数据的方法
- java学习之路 之 基本语法-程序流程控制-(if-else)语句练习题