Region Server意外退出之后…

原地址：http://www.spnguru.com/2011/04/region-server%E6%84%8F%E5%A4%96%E9%80%80%E5%87%BA%E4%B9%8B%E5%90%8E/

早晨上班好好的，突然nagios报出一台regionserver挂了。顿时忙碌起来。

上去一看，从log中看到这样一条信息

2011-04-08 04:02:22,083 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expired

之后， regionserver就理直气壮地退出了。

于是查了下代码，看到了在org.apache.hadoop.hbase.regionserver.HRegionSever.java下这样一段代码。

/*** We register ourselves as a watcher on the master address ZNode. This is* called by ZooKeeper when we get an event on that ZNode. When this method* is called it means either our master has died, or a new one has come up.* Either way we need to update our knowledge of the master.* @param event WatchedEvent from ZooKeeper.*/public void process(WatchedEvent event) {EventType type = event.getType();KeeperState state = event.getState();LOG.info(“Got ZooKeeper event, state: ” + state + “, type: ” +type + “, path: ” + event.getPath());// Ignore events if we’re shutting down.if (stopRequested.get()) {LOG.debug(“Ignoring ZooKeeper event while shutting down”);return;}if (state == KeeperState.Expired) {LOG.error(“ZooKeeper session expired”);boolean restart =this.conf.getBoolean(“hbase.regionserver.restart.on.zk.expire”, false);if (restart) {restart();} else {abort();}} else if (type == EventType.NodeDeleted) {watchMasterAddress();} else if (type == EventType.NodeCreated) {getMaster();// ZooKeeper watches are one time only, so we need to re-register our watch.watchMasterAddress();}}

这段注释写的很清楚了。对于一个reigonserver， 他需要将自己注册到Zookeeper上master的Znode上。这样的目的，是当master 宕机或者新的master启动的时候，能及时收到通知。对于regionserver来说，维持和Zookeeper的联系是非常重要的。因为regionserver需要定期的将心跳包发给master server。如果regionserver不能及时的知道master的改变，就会导致regionserver和master失去联系，而成为一个僵死的进程。

于是，在默认情况下，regionserver遇到这种情况，就选择退出。

为什么regionserver 和Zookeeper的session expired? 可能的原因有

1. 网络不好。

2. Java full GC， 这会block所有的线程。如果时间比较长，也会导致session expired.

怎么办？

1. 将Zookeeper的timeout时间加长。

2. 配置“hbase.regionserver.restart.on.zk.expire” 为true。 这样子，遇到ZooKeeper session expired ， regionserver将选择 restart 而不是 abort

具体的配置是，在hbase-site.xml中加入

<property><name>zookeeper.session.timeout</name><value>90000</value><description>ZooKeeper session timeout.HBase passes this to the zk quorum as suggested maximum time for asession.  See http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions“The client sends a requested timeout, the server responds with thetimeout that it can give the client. The current implementationrequires that the timeout be a minimum of 2 times the tickTime(as set in the server configuration) and a maximum of 20 timesthe tickTime.” Set the zk ticktime with hbase.zookeeper.property.tickTime.In milliseconds.</description></property><property><name>hbase.regionserver.restart.on.zk.expire</name><value>true</value><description>Zookeeper session expired will force regionserver exit.Enable this will make the regionserver restart.</description></property>

为了避免java full GC suspend thread 对Zookeeper heartbeat的影响，我们还需要对hbase-env.sh进行配置。

将

export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError \-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"

修改成

export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError \-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled \-XX:+CMSInitiatingOccupancyFraction=70 \-XX:+UseCMSInitiatingOccupancyOnly -XX:+UseParNewGC -Xmn256m"

更多关于Hbase performance tuning 的信息，可以参考

http://wiki.apache.org/hadoop/PerformanceTuning