Kafka连接问题,导致Spark数据分析中断
来源:互联网 发布:淘宝代理怎么发货步骤 编辑:程序博客网 时间:2024/06/04 23:20
Spark Streaming流式计算经常会中断,中断时Kafka日志如下:
[2016-10-08 15:14:02,518] INFO re-registering broker info in ZK for broker 0 (kafka.server.KafkaHealthcheck)
[2016-10-08 15:14:02,531] INFO Registered broker 0 at path /brokers/ids/0 with address 30.209.81.2:9092. (kafka.utils.ZkUtils$)[2016-10-08 15:14:02,531] INFO done re-registering broker (kafka.server.KafkaHealthcheck)
[2016-10-08 15:14:02,532] INFO Subscribing to /brokers/topics path to watch for new topics (kafka.server.KafkaHealthcheck)
[2016-10-08 15:14:02,601] ERROR Closing socket for /30.209.81.2 because of error (kafka.network.Processor)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
at sun.nio.ch.IOUtil.read(IOUtil.java:171)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
at kafka.utils.Utils$.read(Utils.scala:380)
at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:662)
[2016-10-08 15:14:02,738] INFO Closing socket connection to /30.209.81.2. (kafka.network.Processor)
[2016-10-08 15:14:02,800] INFO Closing socket connection to /30.209.81.2. (kafka.network.Processor)
[2016-10-08 15:14:02,908] ERROR Closing socket for /30.209.81.2 because of error (kafka.network.Processor)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
at sun.nio.ch.IOUtil.read(IOUtil.java:171)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
at kafka.utils.Utils$.read(Utils.scala:380)
at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:662)
此时三台ZK的日志如下:
2016-10-08 15:12:33,948 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /30.209.81.2:55750 which had sessionid 0x255de68ef894c0d
2016-10-08 15:12:33,948 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /30.209.81.36:2577
2016-10-08 15:12:33,949 [myid:2] - INFO [Thread-2154098:NIOServerCnxn@1001] - Closed socket connection for client /30.209.81.11:13811 (no session established for client)
2016-10-08 15:12:33,949 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@821] - Processing stat command from /30.209.81.11:13824
2016-10-08 15:12:52,604 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /30.209.81.11:13836
2016-10-08 15:12:52,604 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x55de68ef7f4bcd, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:662)
2016-10-08 15:12:52,605 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /30.209.81.11:13784 which had sessionid 0x55de68ef7f4bcd
2016-10-08 15:12:52,605 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client attempting to establish new session at /30.209.81.36:2577
2016-10-08 15:12:52,605 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /30.209.81.37:33647
2016-10-08 15:12:52,605 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@821] - Processing stat command from /30.209.81.11:13836
2016-10-08 15:12:52,606 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /30.209.81.11:13854
2016-10-08 15:12:52,606 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@832] - Client attempting to renew session 0x155de68ef934fd9 at /30.209.81.37:33647
2016-10-08 15:12:52,606 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@107] - Revalidating client: 0x155de68ef934fd9
2016-10-08 15:12:52,607 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /30.209.81.11:13868
2016-10-08 15:12:52,607 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@821] - Processing stat command from /30.209.81.11:13854
2016-10-08 15:12:52,607 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@588] - Invalid session 0x155de68ef934fd9 for client /30.209.81.37:33647, probably expired
2016-10-08 15:12:52,607 [myid:2] - INFO [Thread-2154100:NIOServerCnxn$StatCommand@655] - Stat command output
2016-10-08 15:12:52,608 [myid:2] - INFO [Thread-2154100:NIOServerCnxn@1001] - Closed socket connection for client /30.209.81.11:13836 (no session established for client)
2016-10-08 15:12:52,610 [myid:2] - INFO [Thread-2154099:NIOServerCnxn$StatCommand@655] - Stat command output
2016-10-08 15:12:52,610 [myid:2] - ERROR [CommitProcessor:2:NIOServerCnxn@180] - Unexpected Exception:
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:153)
at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1076)
at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74)
2016-10-08 15:12:52,618 [myid:2] - INFO [Thread-2154101:NIOServerCnxn$StatCommand@655] - Stat command output
2016-10-08 15:12:52,618 [myid:2] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /30.209.81.11:13881
2016-10-08 15:12:52,619 [myid:2] - INFO [Thread-2154099:NIOServerCnxn@1001] - Closed socket connection for client /30.209.81.11:13824 (no session established for client)
2016-10-08 15:12:52,620 [myid:2] - INFO [CommitProcessor:2:ZooKeeperServer@595] - Established session 0x255de68ef895d5e with negotiated timeout 18000 for client /30.209.81.36:2577
网上有说三台ZK配置不一致导致的:http://community.cloudera.com/t5/CDH-Manual-Installation/FATAL-ha-ZKFailoverController-Unable-to-start-failover/td-p/16000
并通过修改配置解决,但检查本地集群后发现,配置项目,并不存在该问题,遗留问题,待解决。
如果有朋友遇到相同问题且知道解决方法,请留言,谢谢!
- Kafka连接问题,导致Spark数据分析中断
- kafka添加 partion导致 spark 实时任务数据减少
- spark streaming 读取kafka数据问题
- spark-streaming 编程(三)连接kafka消费数据
- spark streaming统计kafka数据计数不准的问题
- spark与kafka连接测试
- spark向kafka写入数据
- mysql wait_timeout设置导致连接池连接失效问题分析
- kafka连接问题
- 应用直接中断连接导致数据被锁(生产故障)
- 从Kafka topic中获取数据并在Spark中进行分析
- Spark集成Kafka源码分析——SparkStreaming从kafak中接收数据
- 使用Flume+Logstash+Kafka+Spark Streaming进行实时日志处理分析【公安大数据】
- GDB分析PHP连接Memcached 导致coredump问题
- kafka问题分析
- 安卓与硬件反复连接,导致的数据问题
- Spark Streaming 与 Kafka 集成分析
- Spark streaming+kafka+logstash日志分析
- HDU-1506-Largest Rectangle in a Histogram(区间DP)
- php 获取图片主要颜色的方法
- [Python初接触]Windows10-32bit+Python2.7.12+OpenCV3.1.0配置
- Intel Code Challenge Final Round (Div. 1 + Div. 2, Combined) B(模拟) && codeforce 724B Batch Sort
- 拷贝别人的东西到自己的工程里,一定注意包名一定要一样!!否则。。。
- Kafka连接问题,导致Spark数据分析中断
- Codeforces Round Intel Code Challenge Final Round D. Dense Subsequence
- Unity-Calculate
- HDOJ 1880 魔咒词典 参考代码
- Intel Code Challenge Final Round (Div. 1 + Div. 2, Combined) -- A. Checking the Calendar(思路)
- 文章标题
- Maven目录结构内容
- 读《<铁人>观后感》有感
- 生活处处有惊喜