Flume 源码学习（二）Channel组件介绍

---

Channel是Flume中第二个组件，是日志从source传输到sink的通道。根据Flume文档，channel有两个重要衡量指标：

• Reliability：可依赖性。即channel需要接收到的event被下个agent接收或被最终的sink接收。Flume使用事务来保证Reliability。支持可以持久化的基于文件系统的FileChannel和不能保证持久化的MemoryChannel
• Recoverability：可恢复性。如果发生Failure，是否能够恢复数据；Flume支持可恢复的FileChannel和不支持恢复但传输速度快得MemoryChannel

---

类图

首先还是来看Channel相关类的类图：

两个望文生义的类的说明：

接口名 作用 BasicChannelSemantics 基本的channel语义的实现，包含了Threadlocal保存的Transaction实例。通俗的讲就是采用ThreadLocal来保存和当前Channel关联的Transaction的抽象父类 SpillableMemoryChannel 集成了FileChannel和MemoryChannel功能，首先用MemoryChannel,在这个channel达到capacity再改用FileChannel继续

MemoryChannel

我们首先来看MemoryChannel的实现，MemoryChannel有几个重要参数可以配置：

• capacity：Channel的最大容积
• transactionCapacity：一个事务最多可以包含多少个events
• byteCapacity：Channel的Queue的最大的byte 容积
• byteCapacityBufferPercent：定义了byteCapacity和评估的event size之间缓冲的百分比，概要的说就是如果MemoryChannel中等待写出的EventList的byte size最多只能达到(1-byteCapacityBufferPercent/100)*100%，剩下的是缓冲buffer（如果没有buffer，很可能出现内存溢出）
• keep-alive：定义了允许在队列中等待的最大秒数

根据上面的类图，我们知道MemoryChannel继承了BasicChannelSemantics抽象类，而后者
有一个抽象方法createTransaction()需要子类自己实现。所以，MemoryChannel中第一个重要的功能就是基于内存的Transaction的实现。MemoryChannel中是MemoryTransaction这一个内部类，代码如下：

private class MemoryTransaction extends BasicTransactionSemantics {    private LinkedBlockingDeque<Event> takeList;//从channel中take，准备推送到sink    private LinkedBlockingDeque<Event> putList;//从source put到channel的队列    private final ChannelCounter channelCounter;//counter    private int putByteCounter = 0;//put的字节counter    private int takeByteCounter = 0;//take的字节counter    public MemoryTransaction(int transCapacity, ChannelCounter counter) {      putList = new LinkedBlockingDeque<Event>(transCapacity);      takeList = new LinkedBlockingDeque<Event>(transCapacity);      channelCounter = counter;    }    //往putList增加Event，如果没有空间抛错    @Override    protected void doPut(Event event) throws InterruptedException {      channelCounter.incrementEventPutAttemptCount();      int eventByteSize = (int)Math.ceil(estimateEventSize(event)/byteCapacitySlotSize);      if (!putList.offer(event)) {        throw new ChannelException(          "Put queue for MemoryTransaction of capacity " +            putList.size() + " full, consider committing more frequently, " +            "increasing capacity or increasing thread count");      }      putByteCounter += eventByteSize;    }    //从外部类MemoryChannel的等待Take的Event List中take一个过来放到事务的takeList里面，作为本次事务需要提交的Event    @Override    protected Event doTake() throws InterruptedException {      channelCounter.incrementEventTakeAttemptCount();      if(takeList.remainingCapacity() == 0) {//首先判断takeList是否还有空间        throw new ChannelException("Take list for MemoryTransaction, capacity " +            takeList.size() + " full, consider committing more frequently, " +            "increasing capacity, or increasing thread count");      }      if(!queueStored.tryAcquire(keepAlive, TimeUnit.SECONDS)) {//通过queueStored这Semaphore信号来判断外部类MemoryChannel是否有queued Event等待被take，用信号量而不是直接判断queueStored（LinkedBlockingDeque<Event>）的poll()主要是后者会阻塞queueStored上的所有操作，尤其是包括为空得情况下put操作。可以提高throughput。        return null;//没有获取到，意味着可能queue当前为空。      }      Event event;      synchronized(queueLock) {        event = queue.poll();//queueStored不为空。      }      Preconditions.checkNotNull(event, "Queue.poll returned NULL despite semaphore " +          "signalling existence of entry");      takeList.put(event);      //计算taken的byte      int eventByteSize = (int)Math.ceil(estimateEventSize(event)/byteCapacitySlotSize);      takeByteCounter += eventByteSize;      return event;    }    //实际事务的提交    @Override    protected void doCommit() throws InterruptedException {      int remainingChange = takeList.size() - putList.size();      if(remainingChange < 0) {//take size < putsize，sink的消费速度慢于source的产生速度        if(!bytesRemaining.tryAcquire(putByteCounter, keepAlive,          TimeUnit.SECONDS)) {//判断是否有足够空间接收putList中events所占的空间          throw new ChannelException("Cannot commit transaction. Byte capacity " +            "allocated to store event body " + byteCapacity * byteCapacitySlotSize +            "reached. Please increase heap space/byte capacity allocated to " +            "the channel as the sinks may not be keeping up with the sources");        }        if(!queueRemaining.tryAcquire(-remainingChange, keepAlive, TimeUnit.SECONDS)) {//判断queue是否还有空间接收（因为生产速度快于消费速度）          bytesRemaining.release(putByteCounter);          throw new ChannelFullException("Space for commit to queue couldn't be acquired." +              " Sinks are likely not keeping up with sources, or the buffer size is too tight");        }      }      int puts = putList.size();//事务期间生产的event      int takes = takeList.size();//事务期间等待消费的event      synchronized(queueLock) {        if(puts > 0 ) {          while(!putList.isEmpty()) {            if(!queue.offer(putList.removeFirst())) {//将putList中新生产的Events保存到queue              throw new RuntimeException("Queue add failed, this shouldn't be able to happen");            }          }        }        putList.clear();//重置事务的putList，下同        takeList.clear();      }      bytesRemaining.release(takeByteCounter);      takeByteCounter = 0;      putByteCounter = 0;      queueStored.release(puts);//从queueStored释放puts个信号量      if(remainingChange > 0) {        queueRemaining.release(remainingChange);//queueRemaining释放remainingChange个信号量      }      if (puts > 0) {        channelCounter.addToEventPutSuccessCount(puts);      }      if (takes > 0) {        channelCounter.addToEventTakeSuccessCount(takes);      }      channelCounter.setChannelSize(queue.size());    }    @Override    protected void doRollback() {      int takes = takeList.size();      synchronized(queueLock) {        Preconditions.checkState(queue.remainingCapacity() >= takeList.size(), "Not enough space in memory channel " +            "queue to rollback takes. This should never happen, please report");        while(!takeList.isEmpty()) {          queue.addFirst(takeList.removeLast());//把takelist中的events原封不动的放回queue        }        putList.clear();      }      bytesRemaining.release(putByteCounter);      putByteCounter = 0;      takeByteCounter = 0;      queueStored.release(takes);      channelCounter.setChannelSize(queue.size());    }  }

通过上面的代码我们可以看到MemoryChanne的createTransaction内部使用的是基于内存的MemoryTransaction，没有实现持久化。在生产和消费的过程中，非常关注byteCapacity，防止内存溢出。

---

FileChannel

接下来来看FileChannel的实现TBD