Flume NG源码分析（六）应用程序使用的RpcClient设计

来源：互联网发布：域名注册人查询编辑：程序博客网时间：2024/06/05 09:30

上一篇Flume NG源码分析（五）使用ThriftSource通过RPC方式收集日志介绍了ThriftSource利用Thrfit服务ThriftSourceProtocol来收集日志。这篇说说flume-ng-sdk中提供给应用层序使用的RpcClient的设计和实现。继续使用ThriftRpcClient来作例子。

先看看ThriftSourceProtocol提供的原生的客户端，它是Thrfit通过flume.thrift文件定义的Thrfit服务默认生成。这个原生的Client提供了网络传输和协议编解码等RPC客户端的基本功能。关于Thrift客户端可以参考这篇Thrift源码分析（三）-- IDL和生成代码分析

public static class Client extends org.apache.thrift.TServiceClient implements Iface {    public static class Factory implements org.apache.thrift.TServiceClientFactory<Client> {      public Factory() {}      public Client getClient(org.apache.thrift.protocol.TProtocol prot) {        return new Client(prot);      }      public Client getClient(org.apache.thrift.protocol.TProtocol iprot, org.apache.thrift.protocol.TProtocol oprot) {        return new Client(iprot, oprot);      }    }    public Client(org.apache.thrift.protocol.TProtocol prot)    {      super(prot, prot);    }    public Client(org.apache.thrift.protocol.TProtocol iprot, org.apache.thrift.protocol.TProtocol oprot) {      super(iprot, oprot);    }    public Status append(ThriftFlumeEvent event) throws org.apache.thrift.TException    {      send_append(event);      return recv_append();    }    public void send_append(ThriftFlumeEvent event) throws org.apache.thrift.TException    {      append_args args = new append_args();      args.setEvent(event);      sendBase("append", args);    }    public Status recv_append() throws org.apache.thrift.TException    {      append_result result = new append_result();      receiveBase(result, "append");      if (result.isSetSuccess()) {        return result.success;      }      throw new org.apache.thrift.TApplicationException(org.apache.thrift.TApplicationException.MISSING_RESULT, "append failed: unknown result");    }    public Status appendBatch(List<ThriftFlumeEvent> events) throws org.apache.thrift.TException    {      send_appendBatch(events);      return recv_appendBatch();    }    public void send_appendBatch(List<ThriftFlumeEvent> events) throws org.apache.thrift.TException    {      appendBatch_args args = new appendBatch_args();      args.setEvents(events);      sendBase("appendBatch", args);    }    public Status recv_appendBatch() throws org.apache.thrift.TException    {      appendBatch_result result = new appendBatch_result();      receiveBase(result, "appendBatch");      if (result.isSetSuccess()) {        return result.success;      }      throw new org.apache.thrift.TApplicationException(org.apache.thrift.TApplicationException.MISSING_RESULT, "appendBatch failed: unknown result");    }  }

来看看Flume NG是如何封装Thrift客户端的。Flume NG支持Avro，Thrfit等多种RPC实现，它的RpcClient层次结构如下

RpcClient接口定义了给应用程序使用的RPC客户端的基本功能

public interface RpcClient {  public int getBatchSize();    public void append(Event event) throws EventDeliveryException;    public void appendBatch(List<Event> events) throws      EventDeliveryException;  public boolean isActive();    public void close() throws FlumeException;}

AbstractRpcClient抽象类实现了RPCClient接口，提供了getBatchSize的默认实现，并增加了configure接口来支持配置

public abstract class AbstractRpcClient implements RpcClient {  protected int batchSize =      RpcClientConfigurationConstants.DEFAULT_BATCH_SIZE;  protected long connectTimeout =      RpcClientConfigurationConstants.DEFAULT_CONNECT_TIMEOUT_MILLIS;  protected long requestTimeout =      RpcClientConfigurationConstants.DEFAULT_REQUEST_TIMEOUT_MILLIS;  @Override  public int getBatchSize(){    return batchSize;  }  @Override  public abstract void append(Event event) throws EventDeliveryException;  @Override  public abstract void appendBatch(List<Event> events)      throws EventDeliveryException;  @Override  public abstract boolean isActive();  @Override  public abstract void close() throws FlumeException;  protected abstract void configure(Properties properties)      throws FlumeException;}

对于一个设计良好的服务框架的客户端来说，有几个基本的特性

1. 服务寻址

2. 连接池管理

3. 客户端实现RPC调用的负载均衡

4. 缓存

5. 容灾处理，失效转移

我们来看看Flume NG是如何设计它的服务客户端的。基本的组件如下：

服务寻址

Flume NG的RPC客户端的服务寻址实现比较简单，只是在Properties配置文件里设置Thrift服务器的IP和端口，然后用这个值来创建TSocket。这里是一个可以扩展点，使服务寻址的能力更强，更灵活

      HostInfo host = HostInfo.getHostInfoList(properties).get(0);      hostname = host.getHostName();      port = host.getPortNumber();// ClientWrapper      public ClientWrapper() throws Exception{      // 使用hostname, port来构建TSocket      transport = new TFastFramedTransport(new TSocket(hostname, port));      transport.open();      client = new ThriftSourceProtocol.Client(new TCompactProtocol        (transport));      // Not a great hash code, but since this class is immutable and there      // is at most one instance of the components of this class,      // this works fine [If the objects are equal, hash code is the same]      hashCode = random.nextInt();    }

连接池管理

首先是使用ClientWrapper类来封装Thrift生成的原生的Client，可以通过Properties配置来设置Client的值，设置socket连接和protocol编解码协议

private class ClientWrapper {    public final ThriftSourceProtocol.Client client;    public final TFastFramedTransport transport;    private final int hashCode;    public ClientWrapper() throws Exception{      transport = new TFastFramedTransport(new TSocket(hostname, port));      transport.open();      client = new ThriftSourceProtocol.Client(new TCompactProtocol        (transport));      // Not a great hash code, but since this class is immutable and there      // is at most one instance of the components of this class,      // this works fine [If the objects are equal, hash code is the same]      hashCode = random.nextInt();    }}

ConnectionPoolManager实现了一个简单的连接池管理类，提供了checkOut和checkIn两个方法来借出和归还连接对象ClientWrapper。使用ReentrantLock和它的条件队列Condition来实现管程的功能，自管理同步操作。当availableClients为空，并且已经达到连接池的最大值时，checkOut操作会阻塞。当checkIn归还连接对象时，唤醒在checkOut上阻塞的线程。

 private class ConnectionPoolManager {    private final Queue<ClientWrapper> availableClients;    private final Set<ClientWrapper> checkedOutClients;    private final int maxPoolSize;    private int currentPoolSize;    private final Lock poolLock;    private final Condition availableClientsCondition;    public ConnectionPoolManager(int poolSize) {      this.maxPoolSize = poolSize;      availableClients = new LinkedList<ClientWrapper>();      checkedOutClients = new HashSet<ClientWrapper>();      poolLock = new ReentrantLock();      availableClientsCondition = poolLock.newCondition();      currentPoolSize = 0;    }    public ClientWrapper checkout() throws Exception {      ClientWrapper ret = null;      poolLock.lock();      try {        if (availableClients.isEmpty() && currentPoolSize < maxPoolSize) {          ret = new ClientWrapper();          currentPoolSize++;          checkedOutClients.add(ret);          return ret;        }        while (availableClients.isEmpty()) {          availableClientsCondition.await();        }        ret = availableClients.poll();        checkedOutClients.add(ret);      } finally {        poolLock.unlock();      }      return ret;    }    public void checkIn(ClientWrapper client) {      poolLock.lock();      try {        availableClients.add(client);        checkedOutClients.remove(client);        availableClientsCondition.signal();      } finally {        poolLock.unlock();      }    }    public void destroy(ClientWrapper client) {      poolLock.lock();      try {        checkedOutClients.remove(client);        currentPoolSize--;      } finally {        poolLock.unlock();      }      client.transport.close();    }    public void closeAll() {      poolLock.lock();      try {        for (ClientWrapper c : availableClients) {          c.transport.close();          currentPoolSize--;        }      /*       * Be cruel and close even the checked out clients. The threads writing       * using these will now get an exception.       */        for (ClientWrapper c : checkedOutClients) {          c.transport.close();          currentPoolSize--;        }      } finally {        poolLock.unlock();      }    }  }}

客户端负载均衡

LoadBalancingRpcClient继承了AbstractRpcClient类，提供了RPC客户端的负载均衡。这是一个装饰器模式的实现。

HostSelector接口定义了负载均衡的接口，它是对HostInfo进行负载均衡，再由HostInfo找到对应的RpcClient对象。

public interface HostSelector {    void setHosts(List<HostInfo> hosts);    Iterator<HostInfo> createHostIterator();    void informFailure(HostInfo failedHost);  }

HostSelector有两个默认的实现

RoundRobinHostSelector是轮询方式的负载均衡实现

RandomOrderHostSelector是随机方式的负载均衡实现

看下RoundRobinHostSelector的实现，它的逻辑主要在OrderSelector这个类中实现

 private static class RoundRobinHostSelector implements HostSelector {    private OrderSelector<HostInfo> selector;    RoundRobinHostSelector(boolean backoff, long maxBackoff){      selector = new RoundRobinOrderSelector<HostInfo>(backoff);      if(maxBackoff != 0){        selector.setMaxTimeOut(maxBackoff);      }    }    @Override    public synchronized Iterator<HostInfo> createHostIterator() {      return selector.createIterator();    }    @Override    public synchronized void setHosts(List<HostInfo> hosts) {      selector.setObjects(hosts);    }    public synchronized void informFailure(HostInfo failedHost){      selector.informFailure(failedHost);    }  }

OrderSelector是一个支持回退backoff算法的顺序选择容器，它的类层次结构如下

父类OrderSelector是抽象类，定义了回退算法，子类RoundRobinOrderSelector和RandomOrderSelector实现了创建迭代器的算法。

RoundRobinOrderSelector的代码如下

1. getIndexList()返回状态正常的对象列表

2. nextHead索引指向当前位置，作为轮询的起点

public class RoundRobinOrderSelector<T> extends OrderSelector<T> {  private int nextHead = 0;  public RoundRobinOrderSelector(boolean shouldBackOff) {    super(shouldBackOff);  }  @Override  public Iterator<T> createIterator() {    List<Integer> activeIndices = getIndexList();    int size = activeIndices.size();    // possible that the size has shrunk so gotta adjust nextHead for that    if (nextHead >= size) {      nextHead = 0;    }    int begin = nextHead++;    if (nextHead == activeIndices.size()) {      nextHead = 0;    }    int[] indexOrder = new int[size];    for (int i = 0; i < size; i++) {      indexOrder[i] = activeIndices.get((begin + i) % size);    }    return new SpecificOrderIterator<T>(indexOrder, getObjects());  }}

对于LoadBalanceRpcClient来说，它的配置文件里，同一个RPC服务的服务器列表至少有两个服务端信息才能使用负载均衡。在配置文件中还配置了回退算法和负载均衡算法相关的配置

 protected void configure(Properties properties) throws FlumeException {    clientMap = new HashMap<String, RpcClient>();    configurationProperties = new Properties();    configurationProperties.putAll(properties);    hosts = HostInfo.getHostInfoList(properties);    if (hosts.size() < 2) {      throw new FlumeException("At least two hosts are required to use the "          + "load balancing RPC client.");    }     String lbTypeName = properties.getProperty(        RpcClientConfigurationConstants.CONFIG_HOST_SELECTOR,        RpcClientConfigurationConstants.HOST_SELECTOR_ROUND_ROBIN);    boolean backoff = Boolean.valueOf(properties.getProperty(            RpcClientConfigurationConstants.CONFIG_BACKOFF,            String.valueOf(false)));    String maxBackoffStr = properties.getProperty(        RpcClientConfigurationConstants.CONFIG_MAX_BACKOFF);    long maxBackoff = 0;    if(maxBackoffStr != null) {      maxBackoff     = Long.parseLong(maxBackoffStr);    }    if (lbTypeName.equalsIgnoreCase(        RpcClientConfigurationConstants.HOST_SELECTOR_ROUND_ROBIN)) {      selector = new RoundRobinHostSelector(backoff, maxBackoff);    } else if (lbTypeName.equalsIgnoreCase(        RpcClientConfigurationConstants.HOST_SELECTOR_RANDOM)) {      selector = new RandomOrderHostSelector(backoff, maxBackoff);    } else {      try {        @SuppressWarnings("unchecked")        Class<? extends HostSelector> klass = (Class<? extends HostSelector>)            Class.forName(lbTypeName);        selector = klass.newInstance();      } catch (Exception ex) {        throw new FlumeException("Unable to instantiate host selector: "            + lbTypeName, ex);      }    }    selector.setHosts(hosts);    isOpen = true;}

客户端负载均衡的主要组件如下

客户端缓存

客户端缓存比较简单，使用了一个Map结构，保存了HostInfo和对应的RPCClient对象，这样可以复用RPCClient对象，这是一个重对象，包含了一个连接池的实例。

clientMap = new HashMap<String, RpcClient>();private synchronized RpcClient getClient(HostInfo info)      throws FlumeException, EventDeliveryException {    throwIfClosed();    String name = info.getReferenceName();    RpcClient client = clientMap.get(name);    if (client == null) {      client = createClient(name);      clientMap.put(name, client);    } else if (!client.isActive()) {      try {        client.close();      } catch (Exception ex) {        LOGGER.warn("Failed to close client for " + info, ex);      }      client = createClient(name);      clientMap.put(name, client);    }    return client;  }

客户端容灾处理

FailoverRpcClient类实现了客户端的容灾处理，它也是装饰器模式的实现，基础了AbstractRpcClient，实现了RpcClient接口FailoverRpcClient主要是实现了失效转移，利用重试机制，当一个RpcClient失效，就使用下一个RpcClient重试RPC请求，直到成功，或者全部失败

FailoverRpcClient也维护了一个HostInfo列表，由HostInfo再找到对应的RpcClient。还维护了一个最大的重试次数maxTries

private synchronized void configureHosts(Properties properties)      throws FlumeException {    if(isActive){      logger.error("This client was already configured, " +          "cannot reconfigure.");      throw new FlumeException("This client was already configured, " +          "cannot reconfigure.");    }    hosts = HostInfo.getHostInfoList(properties);    String tries = properties.getProperty(        RpcClientConfigurationConstants.CONFIG_MAX_ATTEMPTS);    if (tries == null || tries.isEmpty()){      maxTries = hosts.size();    } else {      try {        maxTries = Integer.parseInt(tries);      } catch (NumberFormatException e) {        maxTries = hosts.size();      }    }......}

看一下它的append方法，实现了重试机制来做失效转移

public void append(Event event) throws EventDeliveryException {    //Why a local variable rather than just calling getClient()?    //If we get an EventDeliveryException, we need to call close on    //that specific client, getClient in this case, will get us    //the next client - leaving a resource leak.    RpcClient localClient = null;    synchronized (this) {      if (!isActive) {        logger.error("Attempting to append to an already closed client.");        throw new EventDeliveryException(            "Attempting to append to an already closed client.");      }    }    // Sit in an infinite loop and try to append!    int tries = 0;    while (tries < maxTries) {      try {        tries++;        localClient = getClient();        localClient.append(event);        return;      } catch (EventDeliveryException e) {        // Could not send event through this client, try to pick another client.        logger.warn("Client failed. Exception follows: ", e);        localClient.close();        localClient = null;      } catch (Exception e2) {        logger.error("Failed to send event: ", e2);        throw new EventDeliveryException(            "Failed to send event. Exception follows: ", e2);      }    }    logger.error("Tried many times, could not send event."    throw new EventDeliveryException("Failed to send the event!");  }

0 0