Zeppelin spark.executor.extraClassPath 和 --driver-class-path 配置冲突问题
来源:互联网 发布:数据采集卡软件 编辑:程序博客网 时间:2024/06/08 01:21
报错如下:
WARN [2017-06-27 15:47:59,777] ({pool-2-thread-2} Logging.scala[logWarning]:66) -
SPARK_CLASSPATH was detected (set to '/home/raini/spark/lib/mysql-connector-java-5.1.38-bin.jar:').This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --driver-class-path to augment the driver classpath
- spark.executor.extraClassPath to augment the executor classpath
WARN [2017-06-27 15:47:59,778] ({pool-2-thread-2} Logging.scala[logWarning]:66) - Setting 'spark.executor.extraClassPath' to '/home/raini/spark/lib/mysql-connector-java-5.1.38-bin.jar:' as a work-around.
ERROR [2017-06-27 15:47:59,780] ({pool-2-thread-2} Logging.scala[logError]:91) - Error initializing SparkContext.
org.apache.spark.SparkException: Found both spark.driver.extraClassPath and SPARK_CLASSPATH. Use only the former.
at org.apache.spark.SparkConf$$anonfun$validateSettings$7$$anonfun$apply$8.apply(SparkConf.scala:543)
at org.apache.spark.SparkConf$$anonfun$validateSettings$7$$anonfun$apply$8.apply(SparkConf.scala:541)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.SparkConf$$anonfun$validateSettings$7.apply(SparkConf.scala:541)
at org.apache.spark.SparkConf$$anonfun$validateSettings$7.apply(SparkConf.scala:529)
解决:
1.提交的作业可以通过加入--driver-class-path参数来设置driver的classpath。
$ bin/spark-submit --master local[2] --driver-class-path lib/mysql-connector-java-5.1.35.jar --class spark.SparkToJDBC ./spark-test_2.10-1.0.jar
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/iteblog/com/mysql-connector-java-5.1.35.jar
所以,删掉一个配置即可,这里删掉了spark配置项:export SPARK_CLASSPATH=...
阅读全文
0 0
- Zeppelin spark.executor.extraClassPath 和 --driver-class-path 配置冲突问题
- spark学习-driver和executor调度
- spark的Driver节点和Executor节点
- spark学习-42-Spark的driver理解和executor理解
- Zeppelin Spark配置GroovySpark
- spark zeppelin 问题集锦
- Spark源码分析之worker节点启动driver和executor
- Hadoop+Spark+Scala+R+PostgreSQL+Zeppelin安装过程-SparkR安装配置和Zeppelin安装配置
- Hadoop+Spark+Scala+R+PostgreSQL+Zeppelin安装过程-SparkR安装配置和Zeppelin安装配置
- spark和zeppelin实践二:安装spark和zeppelin
- Spark学习笔记之-Spark-Standalone下driver和executor分配
- Spark Executor Driver资源调度小结
- Spark Executor Driver资源调度小结
- Spark Executor Driver资源调度小结
- Spark Executor Driver资源调度小结
- Spark源码分析之Driver的分配启动和executor的分配启动
- jmx监控spark executor配置
- Spark & Zeppelin
- 小程序-未读消息
- C# XCOPY命令 预先生成事件命令行”和“后期生成事件命令行”
- 归并排序
- 利用VS的 MFC Class Wizard自动生成COM类的包装类
- 京东面试-0.5像素的边框
- Zeppelin spark.executor.extraClassPath 和 --driver-class-path 配置冲突问题
- composer包开发
- HDU-5115-区间DP
- spring配置数据源的4种方式
- Android 原生下拉刷新SwipeRefreshLayout 与 RecycleLayout 使用
- ipcs ipcrm命令
- vue.js环境搭建
- 运行React Native源码中的例子
- Python: matplotlib模块 源码阅读&理解