LogManager分析

在一个broker上的log都是通过LogManger来管理的，LogManager主要负责日志管理，包括日志创建，日志获取，日志清理，所有的读写操作都要委托的那个日志实例

一 核心字段

logDirs： 日志目录

flushCheckMs： flush检查时间

retentionCheckMs： 日志保留检查时间

scheduler：KafkaScheduler调度器，用于调度线程

brokerState： broker状态

ioThreads： 线程

logs：Pool[TopicAndPartition, Log] 用于管理TopicAndPartition和Log之间的对应关系

dirLocks：FileLock集合这些FileLock用来在文件系统层面为每一个log目录加文件锁。在LogMangaer对象初始化的时候，就会将所有log的目录加锁

recoveryPointCheckpoints：Map[File,OffsetCheckpoint] 类型，用于管理每一个log目录与其下的RecoveryPointCheckpoint文件之间的映射关系。在LogManager对象初始化时，会在每一个log目录下创建一个对应的RecoveryPointCheckpoint文件。这个Map的value是OffsetCheckpoint类型的对象，其中封装了对应log目录下RecoveryPointCheckpoint文件，并提供对RecoveryPointCheckpoint文件的读写操作。RecoveryPointCheckpoint文件则记录了该log目录下所有Log的recoveryPointCheckpoint

logCreationOrDeletionLock：创建或者删除Log时需要加锁进行同步

二 重要方法

2.1 LogManager中的定时任务

startup方法会启动三个线程定期执行执行日志清理，日志刷新和recovery 检查点刷新

defstartup() {
  /* Schedule thecleanup task to delete old logs */
  if(scheduler!= null) {
    info("Starting log cleanup with a period of %d ms.".format(retentionCheckMs))
    scheduler.schedule("kafka-log-retention",
                       cleanupLogs,
                       delay = InitialTaskDelayMs,
                       period = retentionCheckMs,
                       TimeUnit.MILLISECONDS)
    info("Starting log flusher with a default period of %d ms.".format(flushCheckMs))
    scheduler.schedule("kafka-log-flusher",
                       flushDirtyLogs,
                       delay = InitialTaskDelayMs,
                       period = flushCheckMs,
                       TimeUnit.MILLISECONDS)
    scheduler.schedule("kafka-recovery-point-checkpoint",
                       checkpointRecoveryPointOffsets,
                       delay = InitialTaskDelayMs,
                       period = flushCheckpointMs,
                       TimeUnit.MILLISECONDS)
  }
  if(cleanerConfig.enableCleaner)
    cleaner.startup()
}

 

# 清理日志的源码

def cleanupLogs() {  debug("Beginning log cleanup...")  var total = 0  val startMs = time.milliseconds  for(log <- allLogs; if !log.config.compact) {    debug("Garbage collecting '" + log.name + "'")    total += log.deleteOldSegments()  }  debug("Log cleanup completed. " + total + " files deleted in " +                (time.milliseconds - startMs) / 1000 + " seconds")}

 

def deleteOldSegments(): Int = {  // 如果日志清理策略时删除，才会删除，如果是合并，则直接返回  if (!config.delete) return 0  deleteRetenionMsBreachedSegments() + deleteRetentionSizeBreachedSegments()}// 删除超过最长存活时间的segments,也就是过期的segmentprivate def deleteRetenionMsBreachedSegments() : Int = {  if (config.retentionMs < 0) return 0  val startMs = time.milliseconds  deleteOldSegments(startMs - _.largestTimestamp > config.retentionMs)}// 根据log的大小决定是否删除最旧的segmentprivate def deleteRetentionSizeBreachedSegments() : Int = {  if (config.retentionSize < 0 || size < config.retentionSize) return 0  var diff = size - config.retentionSize  def shouldDelete(segment: LogSegment) = {    if (diff - segment.size >= 0) {      diff -= segment.size      true    } else {      false    }  }  deleteOldSegments(shouldDelete)}

 

# 刷新日志的源码

private def flushDirtyLogs() = {  debug("Checking for dirty logs to flush...")  // 遍历(topicAndPartition, log)  for ((topicAndPartition, log) <- logs) {    try {      // 获取现在距离上次日志刷新的时间      val timeSinceLastFlush = time.milliseconds - log.lastFlushTime      debug("Checking if flush is needed on " + topicAndPartition.topic + " flush interval  " + log.config.flushMs +            " last flushed " + log.lastFlushTime + " time since last flush: " + timeSinceLastFlush)      // 如果到期，则开始刷新日志，调用的是Log#flush方法      if(timeSinceLastFlush >= log.config.flushMs)        log.flush    } catch {      case e: Throwable =>        error("Error flushing topic " + topicAndPartition.topic, e)    }  }}

 

# 定时将每一个Log的recoveryPoint写入RecoveryPointCheckpoint文件的源码

def checkpointRecoveryPointOffsets() {  this.logDirs.foreach(checkpointLogsInDir)}private def checkpointLogsInDir(dir: File): Unit = {  val recoveryPoints = this.logsByDir.get(dir.toString)  if (recoveryPoints.isDefined) {    this.recoveryPointCheckpoints(dir).write(recoveryPoints.get.mapValues(_.recoveryPoint))  }}

 

// 先将log目录下所有的Log的recoverPoint写入临时文件，然后再用临时文件替换掉原来的// RecoveryPointCheckpoint文件def write(offsets: Map[TopicAndPartition, Long]) {  lock synchronized {    // write to temp file and then swap with the existing file    val fileOutputStream = new FileOutputStream(tempPath.toFile)    val writer = new BufferedWriter(new OutputStreamWriter(fileOutputStream))    try {      writer.write(CurrentVersion.toString)      writer.newLine()      writer.write(offsets.size.toString)      writer.newLine()      offsets.foreach { case (topicPart, offset) =>        writer.write(s"${topicPart.topic} ${topicPart.partition} $offset")        writer.newLine()      }      writer.flush()      fileOutputStream.getFD().sync()    } catch {      case e: FileNotFoundException =>        if (FileSystems.getDefault.isReadOnly) {          fatal("Halting writes to offset checkpoint file because the underlying file system is inaccessible : ", e)          Runtime.getRuntime.halt(1)        }        throw e    } finally {      writer.close()    }    Utils.atomicMoveWithFallback(tempPath, path)  }}

 

2.2 日志压缩

对于Log的清理策略，主要是为了避免大量日志占满磁盘的情况，log-retention配置的阀值可以是全局的，也可以是基于topic的。Kafka提供2种方式：

第一种：delete，即如果LogSegment到期了删除之，或者LogSegment+这次要添加的消息 > LogSegment的最大容量则删除最老的的LogSegment

第二种：compact，进行log压缩，也可以有效减少日志文件文件大小，缓解磁盘紧张情况。

在有些场景中，key和对应的value的值是不断变化的，就像数据库的记录一样，可以对同一条记录进行修改，在kafka中表现就是在某一时刻key=value1,可能在后续某一时刻，添加了一个key=value2的消息，如果消费者只关心最新的值，那么日志压缩就显得很有用，如下图所示：

# 如果启用日志压缩，并不会针对active log segment，即active log segment不会参加日志压缩，只是针对只读的log segment，避免active log segment成为热点，既要读写还要压缩

# 日志压缩的过程是通过多个cleaner线程来完成的，所以我们可以通过调整cleaner线程数来并发压缩性能，减少对整个服务器端的影响

# 一般情况下，Log数据量很大，为了避免cleaner线程和其他业务线程长时间竞争CPU，并不会将active log segment之外的其他可读LogSegment在一次压缩中全部处理掉，而是将这些LogSegment分批处理

 

# 每一个log都会被cleanercheckpoint分成clean和dirty2部分，如下图所示：

clean: 表示已经压缩的部分，压缩之后，offset是断断续续的，不是增量递增的

dirty: 表示未压缩部分，offset依然是连续递增的

# 每一个clean线程根据日志文件cleanableRatio值来优先压缩该值较效的log，也就说dirty占整个日志的比率越大优先级越高。

 

# clean线程在选定需要清理的log后，首先为dirty部分消息建立一个<key,key对应的last offset>的映射关系，该映射通过SkimpyOffset

Map维护，然后重新复制LogSegment，只保留SkimpyOffsetMap中记录的消息，抛弃掉其他消息

 

# 经过日志压缩后，日志文件和索引文件会不断减小，cleaner线程还会对相邻的LogSegmeng进行合并，避免出现过小的日志和索引文件

 

LogCleaner持有一个cleaners字段，用于管理CleanerThread，这个类的主要方法都委托给了LogCleanerManager。

LogCleanerManager主要用于负责每一个Log的压缩状态管理以及cleaner checkpoint的维护和更新：

offsetCheckpointFile: cleaner checkpoint要写入的文件

checkpoints：Map[File,OffsetCheckpoint]类型，用来维护数据目录和cleaner-offset-checkpoint文件之间的对应关系

inProgress：HashMap[TopicAndPartition, LogCleaningState]，记录正在进行的清理的TopicAndPartition的压缩状态

pausedCleaningCond： 暂停分区的清理工作

dirtiestLogCleanableRatio：dirty部分占log文件的比例

lock: 保护checkpoints和inProgress集合的锁

 

当开始进行日志压缩的时候会先进入LogCleaningInProgress状态；压缩任务可以被暂停，状态被变为LogCleaningPaused，直到有线程恢复它；压缩任务被中断，则进入LogCleaningAborted状态

# updateCheckpoints方法：

用于修改cleaner-offset-checkpoint

defupdateCheckpoints(dataDir:File, update:Option[(TopicAndPartition,Long)]) {
  inLock(lock) {
    // 获取指定目录cleaner-offset-checkpoint文件
    val checkpoint= checkpoints(dataDir)
    // 更新会对相同的key的value进行覆盖
    val existing= checkpoint.read().filterKeys(logs.keys)++ update
    // 更新cleaner-offset-checkpoint文件
    checkpoint.write(existing)
  }
}

 

# grabFilthiestCompactedLog

选择要清理的日志文件，接下来把它添加到in-progress

//选择要清理的日志文件，接下来把它添加到in-progress
def grabFilthiestCompactedLog(time:Time): Option[LogToClean] = {
  inLock(lock) {
    val now = time.milliseconds
    // 获取全部log的cleaner checkpoint
    val lastClean= allCleanerCheckpoints
    // 过滤掉clean.policy配置项为deleted的log,因为他们不会进行压缩
    val dirtyLogs= logs.filter{
      case (_, log) =>log.config.compact
    }.filterNot {//跳过已经在in-progress集合里的Log
      case (topicAndPartition, _) =>inProgress.contains(topicAndPartition)// skip anylogs already in-progress
    }.map {
      case (topicAndPartition,log) => // 为每一个日志创建一个对应的LogToClean实例
        // 获取log中可以进行clean的dirty offset和不能进行的dirty offset
        val (firstDirtyOffset,firstUncleanableDirtyOffset) = LogCleanerManager.cleanableOffsets(log,topicAndPartition,
          lastClean, now)
        LogToClean(topicAndPartition,log, firstDirtyOffset,firstUncleanableDirtyOffset)
    }.filter(ltc=> ltc.totalBytes> 0) // skip any empty logs
    // 获取dirtyLogs集合中cleanableRatio最大值的集合,可以理解为清理工作做得最多的比例
    this.dirtiestLogCleanableRatio= if (dirtyLogs.nonEmpty)dirtyLogs.max.cleanableRatioelse 0
    // 过滤掉cleanableRatio小于配置的minCleanableRatio的log
    val cleanableLogs= dirtyLogs.filter(ltc=> ltc.cleanableRatio> ltc.log.config.minCleanableRatio)
    if(cleanableLogs.isEmpty) {
      None
    } else {
      val filthiest= cleanableLogs.max// 选择要压缩的log
      // 添加或更新此分区对应的状态
      inProgress.put(filthiest.topicPartition,LogCleaningInProgress)
      Some(filthiest)
    }
  }
}

 

Cleaner类时真正进行压缩的工作类：每一个CleanerThread会创建一个Cleaner来负责压缩工作：

private[log]def clean(cleanable:LogToClean): Long = {
  stats.clear()
  info("Beginning cleaning of log %s.".format(cleanable.log.name))
  // 获取要清理的log
  val log= cleanable.log
  info("Building offset map for %s...".format(cleanable.log.name))
  // 确定日志压缩最大上限的offset
  val upperBoundOffset= cleanable.firstUncleanableOffset
  // 构建一个key_hash -> offset的映射,就是遍历LogSegment,并填充到OffsetMap
  // 在填充OffsetMap的过程中，之后追加的log中消息的offset会覆盖之前的相同key的offset
  // Offset填满后，就确定了此次日志压缩的结束位置endOffset
  buildOffsetMap(log,cleanable.firstDirtyOffset,upperBoundOffset, offsetMap)
  // 获取key_hash -> offset的映射最后一个offset
  val endOffset= offsetMap.latestOffset+ 1
  stats.indexDone()
 
  // figure outthe timestamp below which it is safe to remove delete tombstones
  // this position is defined to be aconfigurable time beneath the last modified time of the last clean segment
  // 计算科科员安全删除 '删除标志'的LogSegmnet
  val deleteHorizonMs=
    log.logSegments(0,cleanable.firstDirtyOffset).lastOptionmatch {
      case None => 0L
      case Some(seg) =>seg.lastModified - log.config.deleteRetentionMs
  }
  // 决定要被clean的日志的时间戳上限
  val cleanableHorizionMs= log.logSegments(0,cleanable.firstUncleanableOffset).lastOption.map(_.lastModified).getOrElse(0L)

  // 对要压缩的segment进行分组，分组进行clean,分组是按照LogSegment来分的
  info("Cleaning log %s (cleaning prior to %s, discarding tombstones priorto %s)...".format(log.name,new Date(cleanableHorizionMs),new Date(deleteHorizonMs)))
  for (group <- groupSegmentsBySize(log.logSegments(0,endOffset), log.config.segmentSize,log.config.maxIndexSize))
    cleanSegments(log, group,offsetMap, deleteHorizonMs)

  // recordbuffer utilization
  stats.bufferUtilization= offsetMap.utilization
 
  stats.allDone()

  endOffset
}

 

buildOffsetMap：将消息的key和offset添加到OffsetMap中,并返回full是否已经填满的状态

private[log]def buildOffsetMap(log:Log, start: Long,end: Long, map:OffsetMap) {
  map.clear()
  // 从start到end所有的LogSegment
  val dirty= log.logSegments(start,end).toBuffer
  info("Building offset map for log %s for %d segments in offset range [%d,%d).".format(log.name,dirty.size,start, end))
 
  // Add all thecleanable dirty segments. We must take at least map.slots * load_factor,
  // but we may be able to fit more (ifthere is lots of duplication in the dirty section of the log)
  var full= false //判断offset是否被填满了
  // 循环遍历dirty集合，条件是offsetMap未被填满
  for (segment <- dirty if !full) {
    // 检查LogCleanerManager记录的该分区的压缩状态
    checkDone(log.topicAndPartition)
    // 处理单个LogSegment，将消息的key和offset添加到OffsetMap中,并返回full是否已经填满的状态
    full = buildOffsetMapForSegment(log.topicAndPartition, segment,map, start,log.config.maxMessageSize)
    if (full)
      debug("Offset map is full, %d segments fully mapped, segment with baseoffset %d is partially mapped".format(dirty.indexOf(segment), segment.baseOffset))
  }
  info("Offset map for log %s complete.".format(log.name))
}

 

  private def buildOffsetMapForSegment(topicAndPartition:TopicAndPartition, segment: LogSegment, map: OffsetMap, start: Long, maxLogMessageSize: Int): Boolean = {
    var position = segment.index.lookup(start).position
    val maxDesiredMapSize= (map.slots* this.dupBufferLoadFactor).toInt
    while (position< segment.log.sizeInBytes) {//遍历LogSegment
      checkDone(topicAndPartition)// 检查压缩状态
      readBuffer.clear()
      // 从LogSegment中读取消息
      val messages= new ByteBufferMessageSet(segment.log.readInto(readBuffer,position))
      throttler.maybeThrottle(messages.sizeInBytes)
      val startPosition= position
      for (entry <- messages) {
        val message= entry.message
        if (message.hasKey&& entry.offset >= start) {// 只处理有key的消息
          if (map.size< maxDesiredMapSize)
            // 将key和 offset放入OffsetMap中
            map.put(message.key, entry.offset)
          else
            return true // 表示填满了返回
        }
        stats.indexMessagesRead(1)
      }
      position += messages.validBytes //移动position,准备下一次读取
      stats.indexBytesRead(messages.validBytes)

      // if wedidn't read even one complete message, our read buffer may be too small
      if(position== startPosition) // position没有移动，表是没有读取到一个完整的消息，对readbuffer进行扩容重新读取
        growBuffers(maxLogMessageSize)
    }
    restoreBuffers()
    return false
  }
}

 

cleanSegments：分组压缩

private[log]def cleanSegments(log:Log, segments:Seq[LogSegment],map: OffsetMap,deleteHorizonMs: Long) {
  // 创建一个'.cleaned'后缀的日志文件和Offset索引文件和Time索引文件
  val logFile= new File(segments.head.log.file.getPath+ Log.CleanedFileSuffix)
  logFile.delete()
  val indexFile = new File(segments.head.index.file.getPath+ Log.CleanedFileSuffix)
  val timeIndexFile= new File(segments.head.timeIndex.file.getPath+ Log.CleanedFileSuffix)
  indexFile.delete()
  timeIndexFile.delete()
  // 创建对应的FileMessageSet对象 OffsetIndex对象 TimeIndex对象和LogSegment对象
  val messages= new FileMessageSet(logFile,fileAlreadyExists = false, initFileSize = log.initFileSize(), preallocate = log.config.preallocate)
  val index = new OffsetIndex(indexFile,segments.head.baseOffset,segments.head.index.maxIndexSize)
  val timeIndex = new TimeIndex(timeIndexFile,segments.head.baseOffset,segments.head.timeIndex.maxIndexSize)
  val cleaned = new LogSegment(messages,index, timeIndex,segments.head.baseOffset,segments.head.indexIntervalBytes,log.config.randomSegmentJitter,time)

  try {
    // 遍历每一个组
    for (old <- segments) {
      // 判定此LogSegment中的删除标记是否可以安全删除
      val retainDeletes= old.lastModified > deleteHorizonMs
      info("Cleaning segment %s in log %s (largest timestamp %s) into %s, %sdeletes."
          .format(old.baseOffset,log.name,new Date(old.largestTimestamp),cleaned.baseOffset,if(retainDeletes)"retaining" else "discarding"))
      // 进行日志压缩操做
      cleanInto(log.topicAndPartition, old,cleaned, map,retainDeletes, log.config.maxMessageSize)
    }

    // 截断多余的索引项
    index.trimToValidSize()

    // 当LogSegment变成inactive segment，添加最大的即最后的time index entry到time index，
    // 这个entry将会在决定是否删除segment的时候被使用
    cleaned.onBecomeInactiveSegment()

    // trim timeindex
    timeIndex.trimToValidSize()

    // 执行fluash操作，刷新到磁盘
    cleaned.flush()

    // 更新最后修改时间
    val modified= segments.last.lastModified
    cleaned.lastModified= modified

    //log.replaceSegments首先会将文件后缀'.cleaned'改为'.swap',b并将'.cleaned'对象加入跳跃表管理
    // 之后，将分组中LogSegment从segments中删除，最后将文件的'.swap'删除
    info("Swapping in cleaned segment %d for segment(s) %s in log %s.".format(cleaned.baseOffset,segments.map(_.baseOffset).mkString(","),log.name))
    log.replaceSegments(cleaned,segments)
  } catch {
    case e:LogCleaningAbortedException =>
      cleaned.delete()
      throw e
  }
}

 

private[log]def cleanInto(topicAndPartition:TopicAndPartition, source: LogSegment, dest: LogSegment,
    map: OffsetMap,retainDeletes: Boolean, maxLogMessageSize: Int) {
  var position = 0
  while (position< source.log.sizeInBytes) {//遍历待压缩的LogSegment
    checkDone(topicAndPartition)// 检测压缩状态
    // read a chunk of messages and copyany that are to be retained to the write buffer to be written out
    readBuffer.clear()
    writeBuffer.clear()
    var maxTimestamp= Message.NoTimestamp
    var offsetOfMaxTimestamp= -1L
    // 读取消息
    val messages= new ByteBufferMessageSet(source.log.readInto(readBuffer,position))
    throttler.maybeThrottle(messages.sizeInBytes)

    var messagesRead= 0
    // 遍历每一个消息，检测他是否应该被保留
    for (shallowMessageAndOffset <-messages.shallowIterator) {
      val shallowMessage= shallowMessageAndOffset.message
      val shallowOffset= shallowMessageAndOffset.offset
      val size= MessageSet.entrySize(shallowMessageAndOffset.message)
      /*
       * shouldRetainMessage: 判断保留的依据
       * 1 Offset是否包含相同key且offset更大的消息
       * 2 此消息是'删除标记'的且这个LogSegment中的删除标记可以安全删除
       */
      stats.readMessage(size)
      if (shallowMessage.compressionCodec== NoCompressionCodec) {
        if (shouldRetainMessage(source,map, retainDeletes, shallowMessageAndOffset)) {
          // 将需要保留的信息写入writebuffer
          ByteBufferMessageSet.writeMessage(writeBuffer,shallowMessage, shallowOffset)
          stats.recopyMessage(size)
          if (shallowMessage.timestamp> maxTimestamp) {
            maxTimestamp = shallowMessage.timestamp
            offsetOfMaxTimestamp= shallowOffset
          }
        }
        messagesRead += 1
      } else {
        /*
         * 对于压缩消息,使用深层迭代器迭代内部消息，逻辑与非压缩类型相似
         * 如果整个外层消息都需要保留，则不需要重新压缩，直接将整个外层消息写入WriteBuffer
         * 如果一个外层消息中，只有部分内层消息需要保留，需要对保留的内层重新压缩后写入WriteBuffer
         */
        var writeOriginalMessageSet= true
        val retainedMessages= new mutable.ArrayBuffer[MessageAndOffset]
        val shallowMagic= shallowMessage.magic

        for (deepMessageAndOffset <- ByteBufferMessageSet.deepIterator(shallowMessageAndOffset)) {
          messagesRead += 1
          if (shouldRetainMessage(source,map, retainDeletes, deepMessageAndOffset)) {
            // Check forlog corruption due to KAFKA-4298. If we find it, make sure that we overwrite
            // the corrupted entry withcorrect data.
            if (shallowMagic!= deepMessageAndOffset.message.magic)
              writeOriginalMessageSet= false

            retainedMessages+=deepMessageAndOffset
            // We need themax timestamp and message offset for time index
            if (deepMessageAndOffset.message.timestamp> maxTimestamp) {
              maxTimestamp = deepMessageAndOffset.message.timestamp
              offsetOfMaxTimestamp= deepMessageAndOffset.offset
            }
          } else {
            writeOriginalMessageSet= false
          }
        }
        // There areno messages compacted out and no message format conversion, write the originalmessage set back
        if (writeOriginalMessageSet)
          ByteBufferMessageSet.writeMessage(writeBuffer,shallowMessage, shallowOffset)
        else
          compressMessages(writeBuffer,shallowMessage.compressionCodec,retainedMessages)
      }
    }

    position += messages.validBytes
    // 如果有需要保留的消息则将其追加导压缩后的LogSegment
    if (writeBuffer.position> 0) {
      writeBuffer.flip()
      val retained= new ByteBufferMessageSet(writeBuffer)
      dest.append(firstOffset= retained.head.offset,largestTimestamp = maxTimestamp,
        offsetOfLargestTimestamp= offsetOfMaxTimestamp, messages = retained)
      throttler.maybeThrottle(writeBuffer.limit)
    }
   
    // 未读到一个完整的消息，表示readbuffer过下，需要扩容
    if (readBuffer.limit> 0 && messagesRead== 0)
      growBuffers(maxLogMessageSize)
  }
  restoreBuffers() //重置readbuffer和writebuffer
}