windows运行 HiveContext 报错

 

  装在 windows 的 ideal 运行  val sqlContext = new HiveContext(sc)   报 错

Caused by: java.lang.NullPointerException

at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010)

at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)

at org.apache.hadoop.util.Shell.run(Shell.java:455)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)

at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)

at org.apache.hadoop.util.Shell.execCommand(Shell.java:774)

at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:646)

at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:434)

at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:281)

at org.apache.hadoop.hive.ql.session.SessionState.createPath(SessionState.java:639)

at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:567)

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)

... 17 more

 

 

Exception in thread "main" java.lang.RuntimeException: java.lang.NullPointerException

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)

at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:204)

at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)

at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)

at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)

at org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:462)

at org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:461)

at org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)

at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330)

at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)

at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)

at com.ibeifeng.bigdata.spark.sql.SparkSQLHiveTests$.main(SparkSQLHiveTests.scala:38)

at com.ibeifeng.bigdata.spark.sql.SparkSQLHiveTests.main(SparkSQLHiveTests.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)

 

windows本地运行mapreduce

原因分析：错误在 hadoop ，hadoop是为linux机器设置的，当在windows上运行的时候会出现兼容问题，如果在windows运行mapreduce同样会出现这样的问题

解决方法：查看哪里的源码出现问题（不同的机器运行出错的源码不同），然后在源码中找到这些类，在自己的代码中 src/java 目录下建一个和源码相同的包，

把源码copy进去，哪里出错就注释哪里，直到没有错误为止

原理：运行程序的时候先使用项目的代码，再使用源码，项目中的代码把源码的错误覆盖了