hadoop 2.6.0 hadoop 对Namenode image文件和Edit文件进行清理的NNStorageRetentionManager源代码分析

来源：互联网发布：淘宝网帽子冬季编辑：程序博客网时间：2024/06/06 00:36

NNStorageRetentionManager对文件进行定期旋转删除。

NNStorageRetentionManager的构造方法如下:

  public NNStorageRetentionManager(Configuration conf, NNStorage storage,      LogsPurgeable purgeableLogs) {    this(conf, storage, purgeableLogs, new DeletionStoragePurger());  }

构造NNStorageRetentionManager对象的时候，生成一个DeletionStoragePurger对象，此对象有purgeImage(FSImageFile)来清理Image文件，purgeLog(EditLogFile)来清理EditLog文件，在deleteOrWarn方法里，如果删除不成功，会打印一个警告日志，代码如下：

static class DeletionStoragePurger implements StoragePurger {    @Override    public void purgeLog(EditLogFile log) {      LOG.info("Purging old edit log " + log);      deleteOrWarn(log.getFile());    }    @Override    public void purgeImage(FSImageFile image) {      LOG.info("Purging old image " + image);      deleteOrWarn(image.getFile());      deleteOrWarn(MD5FileUtils.getDigestFileForFile(image.getFile()));    }    private static void deleteOrWarn(File file) {      if (!file.delete()) {        // It's OK if we fail to delete something -- we'll catch it        // next time we swing through this directory.        LOG.warn("Could not delete " + file);      }          }  }

DeletionStoragePurger实现了StoragePurger方法，

  static interface StoragePurger {    void purgeLog(EditLogFile log);    void purgeImage(FSImageFile image);  }

NNStorageRetentionManager的构造方法又调用了以下构造方法：

public NNStorageRetentionManager(      Configuration conf,      NNStorage storage,      LogsPurgeable purgeableLogs,      StoragePurger purger) {    this.numCheckpointsToRetain = conf.getInt(        DFSConfigKeys.DFS_NAMENODE_NUM_CHECKPOINTS_RETAINED_KEY,        DFSConfigKeys.DFS_NAMENODE_NUM_CHECKPOINTS_RETAINED_DEFAULT);    this.numExtraEditsToRetain = conf.getLong(        DFSConfigKeys.DFS_NAMENODE_NUM_EXTRA_EDITS_RETAINED_KEY,        DFSConfigKeys.DFS_NAMENODE_NUM_EXTRA_EDITS_RETAINED_DEFAULT);    this.maxExtraEditsSegmentsToRetain = conf.getInt(        DFSConfigKeys.DFS_NAMENODE_MAX_EXTRA_EDITS_SEGMENTS_RETAINED_KEY,        DFSConfigKeys.DFS_NAMENODE_MAX_EXTRA_EDITS_SEGMENTS_RETAINED_DEFAULT);    Preconditions.checkArgument(numCheckpointsToRetain > 0,        "Must retain at least one checkpoint");    Preconditions.checkArgument(numExtraEditsToRetain >= 0,        DFSConfigKeys.DFS_NAMENODE_NUM_EXTRA_EDITS_RETAINED_KEY +        " must not be negative");        this.storage = storage;    this.purgeableLogs = purgeableLogs;    this.purger = purger;  }

在每次调用saveFSImageInAllDirs的时候，就会调用一次purgeOldStorage()方法。

0 0