Spark错误异常-sparkMaster GC overhead limit exceeded
来源:互联网 发布:网络教学app有哪些 编辑:程序博客网 时间:2024/05/17 20:33
Spark在以standalone模式运行一段时间户总会出现Spark Master GC overhead limit exceeded异常
16/09/20 05:42:24 ERROR ActorSystemImpl: Uncaught fatal error from thread [sparkMaster-akka.actor.default-dispatcher-6] shutting down ActorSystem [sparkMaster]java.lang.OutOfMemoryError: GC overhead limit exceeded
Spark master的内存被GC释放不掉,主要是因为随着时间的推移会缓存大量的对象Web UI中需要显示的已完成的Applications对象,默认设置的缓存数据是50,而启动的spark master进程内存默认设置的大小1G -Xms1g -Xmx1g
spark.history.retainedApplications50在配置文件spark-default.conf中修改保存为一个合适的数量spark.eventLog.enabled truespark.eventLog.dir hdfs://namespace/tmp/spark/eventsspark.serializer org.apache.spark.serializer.KryoSerializerspark.history.fs.logDirectory hdfs://namespace/tmp/spark/eventsspark.history.ui.port 18080spark.history.retainedApplications 20spark.kafka.metadata.broker.list kafka1:9092,kafka2:9092,kafka3:9092spark.flume.listener.port 44445spark.executor.extraJavaOptions -XX:HeapDumpPath=/data0/spark/temp/dump
0 0
- Spark错误异常-sparkMaster GC overhead limit exceeded
- GC overhead limit exceeded : Spark
- spark-OutOfMemory:GC overhead limit exceeded 解决
- spark OutOfMemoryError (GC overhead limit exceeded)
- GC overhead limit exceeded 异常处理
- OOM(GC overhead limit exceeded)异常
- eclipse错误:GC overhead limit exceeded
- gc overhead limit exceeded eclipse错误解决方案
- gc overhead limit exceeded eclipse错误解决方案
- Myeclipse错误:GC overhead limit exceeded
- GC overhead limit exceeded
- GC overhead limit exceeded
- GC overhead limit exceeded
- spark出现GC overhead limit exceeded和java heap space
- spark出现GC overhead limit exceeded和java heap space
- spark-OutOfMemory:GC overhead limit exceeded 解决,timelimitexceeded
- gc overhead limit exceeded eclipse
- gc overhead limit exceeded解决方案
- 第7章 sed、gawk介绍与正则表达式-----------(扩展的正则表达式)
- vim中使用鼠标
- 0000 0000 、Hadoop 环境搭建
- 蓝桥杯入门训练
- 390. Elimination Game
- Spark错误异常-sparkMaster GC overhead limit exceeded
- vue 2.0 需注意事项
- 自定义view的正确姿势-教你一步步画出笑脸
- java日志处理组件log4j--log4j.xml配置详解
- unity 3d中 NGUI和UGUI分别是什么?相对于外部UI插件我们要使用哪个呢?
- STM32CubeMX:RTC(万年历方式)
- Android 系列 4.10使用AsyncTask做后台处理
- 七步理解深度学习
- jdk、cglib动态代理