h2o.ai源码解析(2)—启动流程

上一篇中已经给出h2o.ai的整体介绍以及其核心项目h2o.ai/h2o-3的源码目录，本篇给出h2o启动流程的源码分析。启动过程的时序图如下：

下面挑选时序图中的关键接口进行详细功能介绍：

1. 【步骤3】 registerCoreExtensions()加载扩展类

利用Java ServiceLoader的原理加载当前项目目录中所有/resources/META-INF/目录下water.AbstractH2OExtension文件中定义的服务类:（这些服务类都继承了AbstractH2OExtension）。汇总了下加载了扩展类如下：

Project File Content h2o-core water.AbstractH2OExtension water.FailedNodeWatchdogExtension h2o-ext-krbstandalone water.AbstractH2OExtension hex.security.KerberosExtension h2o-ext-xgboost water.AbstractH2OExtension hex.tree.xgboost.XGBoostExtension h2o-grpc water.AbstractH2OExtension ai.h2o.api.GrpcExtension

*2.【步骤5，6】 startLocalNode()启动当前节点和当前cloud并将当前节点作为当前cloud的唯一成员*

/** Initializes the local node and the local cloud with itself as the only member. */  private static void startLocalNode() {    // Figure self out; this is surprisingly hard    NetworkInit.initializeNetworkSockets();    // Do not forget to put SELF into the static configuration (to simulate    // proper multicast behavior)    if( !ARGS.client && STATIC_H2OS != null && !STATIC_H2OS.contains(SELF)) {      Log.warn("Flatfile configuration does not include self: " + SELF+ " but contains " + STATIC_H2OS);      STATIC_H2OS.add(SELF);    }    ......}

其中调用【步骤6】的 initializeNetworkSockets() 初始化启动一个jettyServer加载Web API(默认为ip:host为localhost:54321)

 public static void initializeNetworkSockets( ) {    // Assign initial ports    H2O.API_PORT = H2O.ARGS.port == 0 ? H2O.ARGS.baseport : H2O.ARGS.port;    // Late instantiation of Jetty object, if needed.    if (H2O.getJetty() == null && !H2O.ARGS.disable_web) {      H2O.setJetty(new JettyHTTPD());    }    // API socket is only used to find opened port on given ip.    ServerSocket apiSocket = null;    // At this point we would like to allocate 2 consecutive ports    while (true) {      H2O.H2O_PORT = H2O.API_PORT + 1;      try {        if (!H2O.ARGS.disable_web) {          apiSocket = H2O.ARGS.web_ip == null // Listen to any interface                      ? new ServerSocket(H2O.API_PORT)                      : new ServerSocket(H2O.API_PORT, -1, getInetAddress(H2O.ARGS.web_ip));          apiSocket.setReuseAddress(true);        }        // Bind to the UDP socket        _udpSocket = DatagramChannel.open();        _udpSocket.socket().setReuseAddress(true);        InetSocketAddress isa = new InetSocketAddress(H2O.SELF_ADDRESS, H2O.H2O_PORT);        _udpSocket.socket().bind(isa);        // Bind to the TCP socket also        _tcpSocket = ServerSocketChannel.open();        _tcpSocket.socket().setReceiveBufferSize(water.AutoBuffer.TCP_BUF_SIZ);        _tcpSocket.socket().bind(isa);        // Warning: There is a ip:port race between socket close and starting Jetty        if (!H2O.ARGS.disable_web) {          apiSocket.close();          H2O.getJetty().start(H2O.ARGS.web_ip, H2O.API_PORT);        }        break;      } catch (Exception e) {        ...      }      // Try next available port to bound      H2O.API_PORT += 2;      ...  }

3. 【步骤8,9】initializePersistence()初始化持久化层，当前支持以下四种持久化存储

Key Description ICE 分布式本地磁盘存储 HDFS 可对接后端的hadoop-hdfs集群 S3 Amazon S3对象存储 NFS 标准文件系统

static void initializePersistence() {    _PM = new PersistManager(ICE_ROOT);}

public PersistManager(URI iceRoot) {    I = new Persist[MAX_BACKENDS];    stats = new PersistStatsEntry[MAX_BACKENDS];    for (int i = 0; i < stats.length; i++) {      stats[i] = new PersistStatsEntry();    }    ...    ...    I[Value.ICE ] = ice;    I[Value.NFS ] = new PersistNFS();    try {      Class klass = Class.forName("water.persist.PersistHdfs");      java.lang.reflect.Constructor constructor = klass.getConstructor();      I[Value.HDFS] = (Persist) constructor.newInstance();      Log.info("HDFS subsystem successfully initialized");    }    catch (Throwable ignore) {      Log.info("HDFS subsystem not available");    }    try {      Class klass = Class.forName("water.persist.PersistS3");      java.lang.reflect.Constructor constructor = klass.getConstructor();      I[Value.S3] = (Persist) constructor.newInstance();      Log.info("S3 subsystem successfully initialized");    } catch (Throwable ignore) {      Log.info("S3 subsystem not available");    }  }

4. 【步骤11】startNetworkServices()初始化网络服务，启动UDPReceiver, TCPReceiver, heartbeat, Cleaner(将K/V store数据落到持久化存储中)等网络服务线程。

5. 【步骤14】getAllProviderNames(true)加载数据源解析器
利用ServiceLoader加载当前项目目录中所有/resources/META-INF/目录下water.parser.ParserProvider文件中定义的数据源解析器，支持的数据源类型有以下几种：

Project File Content Source h2o-core water.parser.ParserProvider water.parser.DefaultParserProviders$ArffParserProvider 默认支持的数据格式有ARFF,XSL,CSV，SVMLight（GUESS并不是一种数据格式） water.parser.DefaultParserProviders$XlsParserProvider water.parser.DefaultParserProviders$SVMLightParserProvider water.parser.DefaultParserProviders$CsvParserProvider water.parser.DefaultParserProviders$GuessParserProvider h2o-orc-parser water.parser.ParserProvider water.parser.orc.OrcParserProvider Apache ORC h2o-parquet-parser water.parser.ParserProvider water.parser.parquet.ParquetParserProvider Apache Parquet h2o-avro-parser water.parser.ParserProvider water.parser.avro.AvroParserProvider Apache Avro

【注】：(1)GuessParserProvider并不是解析GUESS格式的源数据，这个解析器的作用是在不知道数据源格式的情况下，根据解析器的优先级依次试着解析源数据
(2)OrcParserProvider默认是不加载的，因为gradle打包时默认不打包h2o-orc-parser模块（参考build.gradle文件中的编译打包处理）

在gradle.properties文件中有如下配置：

6. 【步骤17】registerResourceRoot()加载WEB静态资源
加载h2o-web/src/main/resources/www和h2o-core/src/main/resources/www目录下的静态WEB资源

7.【步骤18】registerRestApiExtensions()注册Rest API资源
(1)利用Java ServiceLoader的原理加载当前项目目录中所有/resources/META-INF/目录下water.api.RestApiExtension文件中定义的REST API注册服务类，这些服务类都继承自AbstractRegister，并重写了registerEndPoints方法注册一系列主要包含httpMethodURI - handlerClass - handlerMethod的RestAPI接口。

Project File Content h2o-ext-xgboost water.api.RestApiExtension hex.api.xgboost.RegisterRestApi h2o-core water.api.RestApiExtension water.api.RegisterV3Api
water.api.RegisterV4Api h2o-automl water.api.RestApiExtension water.automl.RegisterRestApi h2o-algos water.api.RestApiExtension hex.api.RegisterAlgos

(2) 利用Java ServiceLoader的原理加载当前项目目录中所有/resources/META-INF/目录下water.api.Schema中所有的Schema实体类（Schema就是所有Rest API接口需要的POJO）,不再赘述

8.【步骤20,21】startServingRestApi()启动H2O WEB服务