【集群问题解决】Hbase的"Failed deleting my ephemeral node"错误解决
来源:互联网 发布:数据可视化之美 pdf 编辑:程序博客网 时间:2024/06/05 15:28
HBase的一个RegionServer莫名其妙的挂了,看了下日志,还是不明白。
-----------------------------------------------------------------------------------------------------
5:18:27.795 PM WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
Node /hbase/rs/0.0.0.0,60020,1411723106922 already deleted, retry=false
5:18:27.795 PM WARN org.apache.hadoop.hbase.regionserver.HRegionServer
Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/rs/0.0.0.0,60020,1411723106922
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:156)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1265)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1254)
at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1212)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:942)
at java.lang.Thread.run(Thread.java:744)
5:18:27.816 PM INFO org.apache.zookeeper.ZooKeeper
Session: 0x148b13fd9000018 closed
5:18:27.816 PM INFO org.apache.zookeeper.ClientCnxn
EventThread shut down
5:18:27.816 PM INFO org.apache.hadoop.hbase.regionserver.HRegionServer
stopping server null; zookeeper connection closed.
5:18:27.816 PM INFO org.apache.hadoop.hbase.regionserver.HRegionServer
regionserver60020 exiting
5:18:27.816 PM ERROR org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine
Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2320)
-----------------------------------------------------------------------------------------------------
正头大,日志往前翻,有了。
-----------------------------------------------------------------------------------------------------
5:18:27.634 PM FATAL org.apache.hadoop.hbase.regionserver.HRegionServer
Master rejected startup because clock is out of sync
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server bdtest04,60020,1411723106922 has been rejected; Reported time is too far out of sync with master. Time difference of 39663ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:314)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:215)
at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1291)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:5085)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:277)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1935)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:781)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server bdtest04,60020,1411723106922 has been rejected; Reported time is too far out of sync with master. Time difference of 39663ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:314)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:215)
at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1291)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:5085)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1449)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1653)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1711)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:5402)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1933)
... 2 more
5:18:27.650 PM FATAL org.apache.hadoop.hbase.regionserver.HRegionServer
ABORTING region server 0.0.0.0,60020,1411723106922: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server bdtest04,60020,1411723106922 has been rejected; Reported time is too far out of sync with master. Time difference of 39663ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:314)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:215)
at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1291)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:5085)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server bdtest04,60020,1411723106922 has been rejected; Reported time is too far out of sync with master. Time difference of 39663ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:314)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:215)
at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1291)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:5085)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:277)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1935)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:781)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server bdtest04,60020,1411723106922 has been rejected; Reported time is too far out of sync with master. Time difference of 39663ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:314)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:215)
at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1291)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:5085)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2175)
at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1879)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1449)
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1653)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1711)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:5402)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1933)
... 2 more
5:18:27.652 PM FATAL org.apache.hadoop.hbase.regionserver.HRegionServer
RegionServer abort: loaded coprocessors are: []
-----------------------------------------------------------------------------------------------------
很明显,是时间同步上有问题,查了一下RegionServer和Master两台机的时间,果然不一致。
猜测集群机子虽然是对同一台外部服务器进行对时,但是公众的国外对时服务器的原因。
对策就是Master机子对外对时,RegionServer跟Master进行对时。
- 【集群问题解决】Hbase的"Failed deleting my ephemeral node"错误解决
- 【集群问题解决】Hbase的"Failed deleting my ephemeral node"错误解决
- hbase ‘ Failed deleting my ephemeral node’解决
- Failed deleting my ephemeral node
- hadoop+hbase集群错误解决
- SolrCloud: A previous ephemeral live node still exists(sor6.2.1问题解决)
- hbase集群错误集合
- E/JavaBinder:FAILED BINDER TRANSACTION错误解决+RadioGroup调用check方法的时候,onCheckedChanged被调用多次的问题解决
- kafka I wrote this conflicted ephemeral node
- Starting NFS daemon failed的错误解决
- Nginx的connect() failed 错误解决
- HBase启动错误解决
- HBASE伪集群安装、java api、问题解决
- zen-cart 批量插件easy populat错误黄条Deleting of product data debris failed! ...
- HBase的集群搭建
- hbase的集群搭建
- hbase的集群搭建
- Hbase集群的搭建
- Machine Learning With Spark学习笔记(在10万电影数据上训练、使用推荐模型)
- lustre升级
- android上传文件
- css 如何元素设置水平居中
- sql 联级删除 ON DELETE CASCADE
- 【集群问题解决】Hbase的"Failed deleting my ephemeral node"错误解决
- mac上显示全部隐藏文件
- CocoaPods安装和使用教程
- android--RelativeLayout布局的一些理解
- 当感觉已麻木,告诉 Linux 说reboot
- 关于边界值取值的问题
- 机器学习最佳入门学习资料汇总
- 写出高性能的JQuery
- Swift:什么时候使用结构体和类