Job aborted due to stage failure: Total size of serialized results of 31 tasks (1043.8 MB) is bigger
来源:互联网 发布:飞飞cms下载 编辑:程序博客网 时间:2024/06/01 08:19
spark.driver.maxResultSize太小导致的错误
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 31 tasks (1043.8 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1822)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1835)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1848)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1919)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:905)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
at org.apache.spark.rdd.RDD.collect(RDD.scala:904)
at org.apache.spark.rdd.RDD$$anonfun$toArray$1.apply(RDD.scala:930)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
at org.apache.spark.rdd.RDD.toArray(RDD.scala:929)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1822)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1835)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1848)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1919)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:905)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
at org.apache.spark.rdd.RDD.collect(RDD.scala:904)
at org.apache.spark.rdd.RDD$$anonfun$toArray$1.apply(RDD.scala:930)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
at org.apache.spark.rdd.RDD.toArray(RDD.scala:929)
在spark中默认的spark.driver.maxResultSize为1g,在这里我们将val conf =new Config().set("spark.driver.maxResultSize","2g"),也就是这里设置了2g,也可以设置更高。
例如,
val sc = new SparkContext(new SparkConf().setAppName("test").setMaster("local[*]").set("spark.executor.memory","2g").set("spark.driver.maxResultSize", "2g"))
0 0
- Job aborted due to stage failure: Total size of serialized results of 31 tasks (1043.8 MB) is bigger
- Total size of serialized results of 20 tasks (1088.8 MB) is bigger than spark.driver.maxResultSize (
- Spark问题13之Total size of serialized results of 30 tasks (2.0 GB) is bigger than spark.driver.maxResul
- [已解决]Could not allocate CursorWindow '' of size of size 2097152 due to error -12.
- Generation aborted due to errors detected during the verification of the model
- Generation aborted due to errors detected during the verification of the model
- powerpowerdesigner 生成sql文件提示Generation aborted due to errors detected during the verification of the
- 解决:Generation aborted due to errors detected during the verification of the model.
- 【PowerDesigner】:Generation aborted due to errors detected during the verification of the mod
- Scene is unreachable due to lack of entry....
- PowerDesigner15在生成SQL时报错Generation aborted due to errors detected during the verification of the mod
- PowerDesigner15在生成SQL时报错Generation aborted due to errors detected during the verification of the mod
- PowerDesigner15在生成SQL时报错Generation aborted due to errors detected during the verification of the mod
- PowerDesigner15在生成SQL时报错Generation aborted due to errors detected during the verification of the mo
- PowerDesigner在生成SQL时报错Generation aborted due to errors detected during the verification of the mod
- CursorWindow: Could not allocate CursorWindow '......'of size 2097152 due to error -12.
- 出现Could not allocate CursorWindow of size due to error -12.错误的解决方案
- From obscurity to the center of stage
- angular中$state.go页面跳转并传递参数
- Python自定义豆瓣电影种类,排行,点评的爬取与存储(进阶上)
- 几个问题以及对应的解决方案记录
- Spring BeanFactoryPostProcessor和BeanPostProcessor的关系
- Linux学习笔记-001
- Job aborted due to stage failure: Total size of serialized results of 31 tasks (1043.8 MB) is bigger
- UVA 12186 树上dp入门
- 【二进制、字符串】UVA
- 28. Implement strStr()
- 决策树算法优化(三)
- python
- TypeError: unsupported operand type(s) for +: 'int' and 'str'
- 剑指offer——旋转数组的最小数字______
- Pytorch入门学习(三)---- NN包的使用