lease引发的血案

来源：互联网发布：如何使用阿里云服务器编辑：程序博客网时间：2024/04/30 11:32

lease引发的血案

今天线上出现了一个故障惊出一身冷汗，经过查明原来是lease引发的问题。不过查问题的过程让我们耽误了很多修复故障的时间，很是不爽。

起因：datanode和regionserver以及master同时挂掉
现象：datanode重启后，regionserver重启不久，多台regionserver相继即挂掉，log显示：

Html代码

org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append. Requesting close of hlog java.io.IOException: Reflection at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147) at
org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:994) at
org.apache.hadoop.hbase.regionserver.wal.HLog.completeCacheFlush(HLog.java:1176) at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1038) at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:929) at
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:571) at
org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:524) at
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:120) at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151) at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at
java.lang.Thread.run(Thread.java:662) Caused by: java.lang.reflect.InvocationTargetException at
sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at
java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145) ... 11 more
Caused by: java.io.IOException: Error Recovery for block blk_-5430512709521689588_45878056 failed because recovery from primary datanode xx.xx.xx.xx:50010 failed 6 times. Pipeline was xx.xx.xx.xx:50010. Aborting... at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2841) at

org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append. Requesting close of hlog java.io.IOException: Reflection at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:147) at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:994) at org.apache.hadoop.hbase.regionserver.wal.HLog.completeCacheFlush(HLog.java:1176) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1038) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:929) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:571) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:524) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:120) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:145) ... 11 more Caused by: java.io.IOException: Error Recovery for block blk_-5430512709521689588_45878056 failed because recovery from primary datanode xx.xx.xx.xx:50010 failed 6 times. Pipeline was xx.xx.xx.xx:50010. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2841) at

    看起来是datanode的问题，但是登陆datanode发现木有问题。于是再重启regionserver，过一会又报同样的错误退出...
    于是开始查系统的问题。半个小时过去鸟。。。
    实在查不到原因，再重启，发现系统好了。。。

    原因：datanode挂掉的时候，regionserver正在写hlog，这是一个append的过程。当regionserver也挂掉后，则相应的块的client也断开了，很自然这个block连同它的备份都处于需要修复的状态。由于master也挂掉了，所以master被备机接管。接管时候有一步是检查哪些region server处于online状态(waitForRegionServers)。不处于online状态的rs(默认的配置下有一定概率在线的rs也会被判断为不在线，如果hbase.master.wait.on.regionservers.timeout设置为大于6秒则不会出现这种情况)会被master强制执行recoverFileLease。于是引发了namenode对这个block发起recovery block过程，这个过程抢占了lease。当其它region server需要读这个文件或者其它这个datanode原先持有的block的时候都会发现需要recovery block，这个过程由于抢占不到lease导致失败。而写hlog阶段的失败regionserver的处理逻辑是让自己挂掉（这样最安全）。于是会发现凡是需要写这个hlog的region server会连续挂掉。
    虽然regionserver挂掉了，但是它对这个坏块仍然发起过一次写操作，于是这个block上的lease由1小时的硬lease降级为了1分钟的软lease，即1分钟后租约消失。所以1分钟后再次发起请求就恢复正常了。
    但是为什么再次重启regionserver也挂掉了呢？原因是当时datanode上面还有其它正在被写的hlog的block，所以多重启几次就好了。事实上当时如果我们不等待这半小时而是直接手忙脚乱重启的话，故障就会更快恢复了。。。

    结论：master和regionserver不能同时挂掉。只要不是同时挂掉，就不会导致recovery block的发生，也就不会发生lease的杯具了。不过这种情况很难发生，所以我们决定将hbase.master.wait.on.regionservers.timeout参数改为10秒。