hadoop2.2.0 HA中active的namenode死掉了
来源:互联网 发布:哒哒网游加速软件 编辑:程序博客网 时间:2024/04/30 08:23
2014-07-19 21:55:49,823 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 15414 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [192.168.1.202:8485]
2014-07-19 21:56:11,660 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 37251 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [192.168.1.202:8485]
2014-07-19 21:56:22,652 FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: flush failed for required journal (JournalAndStream(mgr=QJM to [192.168.1.200:8485, 192.168.1.201:8485, 192.168.1.202:8485], stream=QuorumOutputStream starting at txid 190))
java.io.IOException: Timed out waiting 20000ms for a quorum of nodes to respond.
at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:137)
at org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.flushAndSync(QuorumOutputStream.java:107)
at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:113)
at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:107)
at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$8.apply(JournalSet.java:492)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:352)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.access$100(JournalSet.java:55)
at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.flush(JournalSet.java:488)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:613)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1057)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:995)
at org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:1082)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:5050)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:832)
at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:11214)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
2014-07-19 21:56:24,074 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Aborting QuorumOutputStream starting at txid 190
2014-07-19 21:56:28,232 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Took 55983ms to send a batch of 1 edits (13 bytes) to remote journal 192.168.1.200:8485
2014-07-19 21:56:30,303 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Took 55896ms to send a batch of 1 edits (13 bytes) to remote journal 192.168.1.201:8485
2014-07-19 21:57:09,300 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2014-07-19 21:57:14,382 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at slave01/192.168.1.200
2014-07-19 21:56:11,660 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 37251 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [192.168.1.202:8485]
2014-07-19 21:56:22,652 FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: flush failed for required journal (JournalAndStream(mgr=QJM to [192.168.1.200:8485, 192.168.1.201:8485, 192.168.1.202:8485], stream=QuorumOutputStream starting at txid 190))
java.io.IOException: Timed out waiting 20000ms for a quorum of nodes to respond.
at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:137)
at org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.flushAndSync(QuorumOutputStream.java:107)
at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:113)
at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:107)
at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$8.apply(JournalSet.java:492)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:352)
at org.apache.hadoop.hdfs.server.namenode.JournalSet.access$100(JournalSet.java:55)
at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.flush(JournalSet.java:488)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:613)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1057)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:995)
at org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:1082)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:5050)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:832)
at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139)
at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:11214)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
2014-07-19 21:56:24,074 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Aborting QuorumOutputStream starting at txid 190
2014-07-19 21:56:28,232 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Took 55983ms to send a batch of 1 edits (13 bytes) to remote journal 192.168.1.200:8485
2014-07-19 21:56:30,303 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Took 55896ms to send a batch of 1 edits (13 bytes) to remote journal 192.168.1.201:8485
2014-07-19 21:57:09,300 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2014-07-19 21:57:14,382 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at slave01/192.168.1.200
************************************************************/
active节点死掉后,没有进行故障转移和切换,standy节点没有自动转变成active状态。导致整个集群死掉。
0 0
- hadoop2.2.0 HA中active的namenode死掉了
- hadoop2.2.0 HA启动时出现了两个standy的Namenode,没有出现active的Namenode
- 解决:hadoop2.5.2 HA启动时出现了两个standy的Namenode,没有出现active的Namenode
- hadoop2 namenode HA的问题
- hadoop2 namonode为HA 得到hadoop的active namenode具体地址代码
- Hadoop2.7.0集群的NameNode在HA下如何切换active和standby状态
- 配置hadoop2.X的namenode HA及Yarn HA
- hadoop2.2.0双namenode配置文件配置(高可靠性HA)
- hadoop2中ResourceManager的HA
- Hadoop 2.0 中 NameNode/ResourceManager HA 总结
- Hadoop2之NameNode—HA原理详解
- hadoop2—namenode—HA原理详解
- hadoop2.x手动切换namenode active
- spark结合Hadoop2.2.0 HA使用中遇到的问题
- hadoop2.x通过Zookeeper来实现namenode的HA方案以及ResourceManager单点故障的解决方案
- hadoop2.x通过Zookeeper来实现namenode的HA方案以及ResourceManager单点故障的解决方案
- 分享下看到的一篇 十分受用的关于hadoop2—namenode—HA原理详解
- HDFS中namenode的HA高可用机制
- Visual F# Power Tools 简介
- BEGINNING SHAREPOINT® 2013 DEVELOPMENT 第1章节--SharePoint 2013 介绍 处理开发人员需求
- SDK Manager 无法更新、下载问题解决
- 深度优先搜索(堆栈)解决走迷宫问题
- SAX 解析详解
- hadoop2.2.0 HA中active的namenode死掉了
- 精通安卓性能优化-第七章(二)
- 更改TreeView的节点名
- Spring 4.0.6 + Hibernate 4.3.5.1.Final + JPA2.0 + DBCP2 集成
- word2vec 中的数学原理详解(四)基于 Hierarchical Softmax 的模型
- HTML学习笔记(6)--列表
- C语言:大数相加与大数相减.
- Spark:一个高效的分布式计算系统
- andorid style 使用与误区