spark出现task不能序列化错误的解决方法 org.apache.spark.SparkException: Task not serializable
来源:互联网 发布:个人简历免费下载软件 编辑:程序博客网 时间:2024/05/22 00:47
出现“task not serializable"这个错误,一般是因为在map、filter等的参数使用了外部的变量,但是这个变量不能序列化。特别是当引用了某个类(经常是当前类)的成员函数或变量时,会导致这个类的所有成员(整个类)都需要支持序列化。解决这个问题最常用的方法有:
- 如果可以,将依赖的变量放到map、filter等的参数内部定义。这样就可以使用不支持序列化的类;
- 如果可以,将依赖的变量独立放到一个小的class中,让这个class支持序列化;这样做可以减少网络传输量,提高效率;
- 如果可以,将被依赖的类中不能序列化的部分使用transient关键字修饰,告诉编译器它不需要序列化。
- 将引用的类做成可序列化的。
- 以下这两个没试过。。
- Make the NotSerializable object as a static and create it once per machine.
- Call rdd.forEachPartition and create the NotSerializable object in there like this:
==================
ref[1]:<http://databricks.gitbooks.io/databricks-spark-knowledge-base/content/troubleshooting/javaionotserializableexception.html>
If you see this error:
org.apache.spark.SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException: ...
The above error can be triggered when you intialize a variable on the driver (master), but then try to use it on one of the workers. In that case, Spark Streaming will try to serialize the object to send it over to the worker, and fail if the object is not serializable. Consider the following code snippet:
NotSerializable notSerializable = new NotSerializable();JavaRDD<String> rdd = sc.textFile("/tmp/myfile");rdd.map(s -> notSerializable.doSomething(s)).collect();
This will trigger that error. Here are some ideas to fix this error:
- Serializable the class
- Declare the instance only within the lambda function passed in map.
- Make the NotSerializable object as a static and create it once per machine.
- Call rdd.forEachPartition and create the NotSerializable object in there like this:
rdd.forEachPartition(iter -> { NotSerializable notSerializable = new NotSerializable(); // ...Now process iter});
Pasted from: <http://databricks.gitbooks.io/databricks-spark-knowledge-base/content/troubleshooting/javaionotserializableexception.html>
另外, stackoverflow上http://stackoverflow.com/questions/25914057/task-not-serializable-exception-while-running-apache-spark-job 这个答的也很简明易懂。
From WizNote
0 0
- spark出现task不能序列化错误的解决方法 org.apache.spark.SparkException: Task not serializable
- spark出现task org.apache.spark.SparkException: Task not serializable
- spark出现“org.apache.spark.SparkException: Task not serializable"
- org.apache.spark.SparkException: Task not serializable
- spark + quartz : org.apache.spark.SparkException: Task not serializable
- Spark[二]:org.apache.spark.SparkException: Task not serializable
- org.apache.spark.SparkException: Task not serializable问题分析
- spark学习-52-Spark的org.apache.spark.SparkException: Task not serializable
- Spark运行程序异常信息: org.apache.spark.SparkException: Task not serializable 解决办法
- spark出现task不能序列化错误的解决方法
- Exception in thread "main" org.apache.spark.SparkException: Task not serializable异常
- Exception in thread "main" org.apache.spark.SparkException: Task not serializable--two
- Spark Q&A : Task/Object not serializable 任务不能序列化
- 针对Apache Spark logging within scala 出现的 Task not serializable
- Spark task not serializable错误的分析和处理
- Spark Task未序列化(Task not serializable)问题分析及解决
- Task not serializable exception while running apache spark job
- spark: Task not serializable (java)
- iOS如何显示桌面应用图标上的推送消息数(大神勿进)
- Martix精确缩放图片:Matrix与Bitmap的使用
- Android百度地图相关总结
- 【Java并发编程】之十六:深入Java内存模型——happen-before规则及其对DCL的分析(含代码)
- ping过程详解
- spark出现task不能序列化错误的解决方法 org.apache.spark.SparkException: Task not serializable
- quartz表达式
- Java:类的访问修饰符
- html中meta标签的作用
- solr 5.3安装
- ORA-00604和ORA-04031导致数据库实例宕机
- flash
- 【Java并发编程】之十七:深入Java内存模型—内存操作规则总结
- JDK,JRE,JVM区别与联系