KafkaConsumer 抛出KafkaConsumer is not safe for multi-threaded
来源:互联网 发布:英雄皮肤淘宝店 编辑:程序博客网 时间:2024/06/05 15:13
KafkaConsumer 抛出KafkaConsumer is not safe for multi-threaded access异常
环境:spark-2.x kafka_0.10.x
异常:
java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1431) at org.apache.kafka.clients.consumer.KafkaConsumer.seek(KafkaConsumer.java:1132) at org.apache.spark.streaming.kafka010.CachedKafkaConsumer.seek(CachedKafkaConsumer.scala:95) at org.apache.spark.streaming.kafka010.CachedKafkaConsumer.get(CachedKafkaConsumer.scala:69) at org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:227) at org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:193) at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:364)at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1021) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:996) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:936) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:996) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:700) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:105) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
异常分析:
当kafka执行rebalance时,kafka可能抛出此异常。confluent存在两种worker: 1.负责数据读写的source/sink worker; 2.负责协调source/sink worker的herder worker;当rebalance发生时,herder会去主动close sink worker线程。如果sink worker正在操作就会抛出异常。当代码中的slideDuration比batchDuration大很多时也会抛出异常:java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access
解决思路:
1.对使用window操作的DStream在调用window之前先调用checkpoint方法,可以截断lineage,从而避免这个问题。2.修改slideDuration和batchDuration,使得两个的值很相近
版权声明:本文为博主原创,未经允许请勿随意转载。
阅读全文
0 0
- KafkaConsumer 抛出KafkaConsumer is not safe for multi-threaded
- KafkaConsumer is not safe for multi-threaded access
- KafkaConsumer分析
- KafkaConsumer及其监控
- Apache Kafka 0.9 KafkaConsumer
- KafkaConsumer流程简述
- KafkaConsumer流程简述
- KafkaConsumer使用详解
- 【Kafka源码】KafkaConsumer
- Kafka 0.10 KafkaConsumer流程简述
- Kafka 0.10 KafkaConsumer流程简述
- Kafka 0.10 KafkaConsumer流程简述
- Kafka 0.10 KafkaConsumer流程简述
- Kafka 0.10 KafkaConsumer流程简述
- Kafka 0.10 KafkaConsumer流程简述
- Kafka 0.10 KafkaConsumer流程简述
- Volatile: Almost Useless for Multi-Threaded Programming
- Fetcher: KafkaConsumer消息消费的管理者
- Spring Boot 构建框架
- 九九乘法表
- 实验楼sql进阶之成绩管理系统的数据操作(window)
- JDBC的批处理操作三种方式 pstmt.addBatch()
- 使用ALT+数字小键盘在文本文件中输入特殊字符
- KafkaConsumer 抛出KafkaConsumer is not safe for multi-threaded
- 在Linux和Windows下安装SVN钩子脚本
- 聚类
- 修改div内A标签的href值
- POJ 1475 Pushing Boxes
- android监听返回按键
- Codeforces Round #442 (Div. 2)-广搜&剪枝&技巧&好题-D. Olya and Energy Drinks
- 线性表
- C# 利用函数反射、XML序列化/反序列化保存函数执行与输入参数列表