org.apache.hadoop.hbase.ClockOutOfSyncException

之前启动hbase一直没事，停了一段时间后重启，发现启动不了了，总是报错，配了3个reg，只启动了一个，有两个因为节点时间差距导致启动失败

2013-10-08 11:53:32,756 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Master rejected startup because clock is out of sync

org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop-sl001.xxx.com,60020,1381204411958 has been rejected; Reported time is too far out of sync with master.  Time difference of 110252ms > max allowed of 30000ms
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2086)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:744)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop-sl001.xxx.com,60020,1381204411958 has been rejected; Reported time is too far out of sync with master.  Time difference of 110252ms > max allowed of 30000ms
        at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:235)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:169)
        at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1067)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)


        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
        at $Proxy8.regionServerStartup(Unknown Source)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2084)
        ... 2 more
2013-10-08 11:53:32,760 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server hadoop-sl001.xxx.com,60020,1381204411958: Unhandled exception: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop-sl001.xxx.com,60020,1381204411958 has been rejected; Reported time is too far out of sync with master.  Time difference of 110252ms > max allowed of 30000ms
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop-sl001.xxx.com,60020,1381204411958 has been rejected; Reported time is too far out of sync with master.  Time difference of 110252ms > max allowed of 30000ms
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2086)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:744)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop-sl001.xxx.com,60020,1381204411958 has been rejected; Reported time is too far out of sync with master.  Time difference of 110252ms > max allowed of 30000ms
        at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:235)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:169)
        at org.apache.hadoop.hbase.master.HMaster.regionServerStartup(HMaster.java:1067)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)


        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:995)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
        at $Proxy8.regionServerStartup(Unknown Source)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2084)
        ... 2 more
2013-10-08 11:53:32,763 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
2013-10-08 11:53:32,763 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to report fatal error to master
java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:1918)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:799)
        at java.lang.Thread.run(Thread.java:662)
2013-10-08 11:53:32,763 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled exception: org.apache.hadoop.hbase.ClockOutOfSyncException: Server hadoop-sl001.xxx.com,60020,1381204411958 has been rejected; Reported time is too far out of sync with master.  Time difference of 110252ms > max allowed of 30000ms

2013-10-08 11:53:32,763 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020

在hbase-site.xml文件中 修改增加 

 <property>
    <name>hbase.master.maxclockskew</name>
    <value>150000</value>
  </property>

重启即可

看http://192.168.232.60:60010/master-status 中

Region Servers有三个了。