Hbase 各节点时间同步

at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)

        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
        at $Proxy8.regionServerStartup(Unknown Source)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1847)
        ... 2 more
2013-01-05 10:43:44,547 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server yfhadoopd3,60020,1357353729598: Unhandled exception: org.apache.hadoop.hbase.ClockOutOfSyncExcep
tion: Server yfhadoopd3,60020,1357353729598 has been rejected; Reported time is too far out of sync with master.  Time difference of 33455ms > max allowed of 30000ms
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server yfhadoopd3,60020,1357353729598 has been rejected; Reported time is too far out of sync with mast
er.  Time difference of 33455ms > max allowed of 30000ms
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1849)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:666)
        at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server yfhadoopd3,60020,1357353729598 has been rejected; Reported time is too far out of sync with mas
ter.  Time difference of 33455ms > max allowed of 30000ms
        at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:214)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:151)
        at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:860)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1376)

        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
        at $Proxy8.regionServerStartup(Unknown Source)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1847)
        ... 2 more
2013-01-05 10:43:44,550 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
2013-01-05 10:43:44,551 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to report fatal error to master
java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:1688)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:718)
        at java.lang.Thread.run(Thread.java:722)
2013-01-05 10:43:44,551 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled exception: org.apache.hadoop.hbase.ClockOutOfSyncException: Server yfhadoopd3,60020,1357353729598 has
 been rejected; Reported time is too far out of sync with master.  Time difference of 33455ms > max allowed of 30000ms
2013-01-05 10:43:44,551 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
2013-01-05 10:43:44,552 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Block cache LRU eviction started; Attempting to free -1080121.52 KB of total=11.67 MB
2013-01-05 10:43:44,553 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server null
2013-01-05 10:43:44,553 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker
2013-01-05 10:43:44,553 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server null; all regions closed.
2013-01-05 10:43:44,554 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver60020 closing leases
2013-01-05 10:43:44,554 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver60020 closed leases
2013-01-05 10:43:44,575 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/rs/yfhadoopd3,60020,1357353729598 already deleted, and this is not a retry
2013-01-05 10:43:44,575 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/rs/yfhadoopd3,60020,1357353729598
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:868)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:112)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:966)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:955)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1006)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:786)
        at java.lang.Thread.run(Thread.java:722)

 

 

才知道原来时间对不上.
同时在几个机器上校正时间:

linux
date -s 08:42:00