HDFS block丢失过多进入安全模式（safe mode）的解决方法

HDFS block丢失过多进入安全模式（Safe mode）的解决方法

背景及现象描述(Background and Symptom)

因磁盘空间不足，内存不足，系统掉电等其他原因导致dataNode datablock丢失，出现如下类似日志：

The number of live datanodes 3 has reached the minimum number 0.Safe mode will be turned off automatically once the thresholds have been reached.Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Log not rolled. Name node is in safe mode.The reported blocks 632758 needs additional 5114 blocks to reach the threshold 0.9990of total blocks 638510.The number of live datanodes 3 has reached the minimum number 0.Safe mode will be turned off automatically once the thresholds have been reached.at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1209)    ... 12 more

原因分析(Cause Analysis)*

由于系统断电，内存不足等原因导致dataNode丢失超过设置的丢失百分比，系统自动进入安全模式

解决办法(Solution)*

安装HDFS客户端，并执行如下命令：

步骤 1     执行命令退出安全模式：

hadoop dfsadmin -safemode leave

步骤 2     执行健康检查，删除损坏掉的block。

hdfs fsck  /  -delete

注意: 这种方式会出现数据丢失，损坏的block会被删掉