storm源码分析-日志服务

来源：互联网发布：越狱iphone抹除数据编辑：程序博客网时间：2024/06/06 05:12

本文为作者原创，转载请标明出处。原作者：Tony_老七

原文链接：http://blog.csdn.net/tonylee0329/article/details/40081723

以下基于storm的版本: 0.9.0.1

storm中各种日志是排查问题的有效工具，但是这些日志是怎么形成的呢？

storm使用logback作为日志服务插件，配置文件见$STORM_HOME/logback/cluster.xml

我们目前主要关心worker、nimbus、ui、supervisor日志，METRICS、ACCESS这些比较简单

worker-xxxx.log

nimbus.log

supervisor.log

ui.log

这些日志都是使用了A1这个appender

<appender name="A1" class="ch.qos.logback.core.rolling.RollingFileAppender">    <file>${storm.home}/logs/${logfile.name}</file>    <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">      <fileNamePattern>${storm.home}/logs/${logfile.name}.%i</fileNamePattern>      <minIndex>1</minIndex>      <maxIndex>9</maxIndex>    </rollingPolicy>    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">      <maxFileSize>100MB</maxFileSize>    </triggeringPolicy>    <encoder>      <pattern>%d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] %m%n</pattern>    </encoder></appender>

采用了logback的日志轮转功能，但是这里的${storm.home}、${logfile.name}这个是在哪定义的呢？

从bin/storm脚本中，可以看到nimbus、ui、supervisor是在启动的时候传入了变量logfile.name

如nimbus启动的时候

def nimbus(klass="backtype.storm.daemon.nimbus"):    """Syntax: [storm nimbus]    Launches the nimbus daemon. This command should be run under    supervision with a tool like daemontools or monit.    See Setting up a Storm cluster for more information.    (https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster)    """    cppaths = [CLUSTER_CONF_DIR]    jvmopts = parse_args(confvalue("nimbus.childopts", cppaths)) + [       <span style="color:#ff0000;"> "-Dlogfile.name=nimbus.log",</span>        "-Dlogback.configurationFile=" + STORM_DIR + "/logback/cluster.xml",    ]      exec_storm_class(        klass,        jvmtype="-server",        extrajars=cppaths,        jvmopts=jvmopts)

worker的日志因为涉及到当前使用端口，并且只有在集群有topology运行的时候才会生成

阅读源码，可以看到在supervisor.clj的launch-worker方法中生成了logfile.name

在启动worker进程的时候，根据worker所在port，形成变量名logfile.name,logback中据此引用变量,形成名称worker-6700.log

(defmethod launch-worker    :distributed [supervisor storm-id port worker-id]    (let [conf (:conf supervisor)          storm-home (System/getProperty "storm.home")          stormroot (supervisor-stormdist-root conf storm-id)          stormjar (supervisor-stormjar-path stormroot)          storm-conf (read-supervisor-storm-conf conf storm-id)          classpath (add-to-classpath (current-classpath) [stormjar])          childopts (.replaceAll (str (conf WORKER-CHILDOPTS) " " (storm-conf TOPOLOGY-WORKER-CHILDOPTS))                                 "%ID%"                                 (str port))          <span style="color:#ff0000;">logfilename (str "worker-" port ".log")</span>          command (str "java -server " childopts                       " -Djava.library.path=" (conf JAVA-LIBRARY-PATH)                       " -Dlogfile.name=" logfilename                       " -Dstorm.home=" storm-home                       " -Dlogback.configurationFile=" storm-home "/logback/cluster.xml"                       " -cp " classpath " backtype.storm.daemon.worker "                       (java.net.URLEncoder/encode storm-id) " " (:assignment-id supervisor)                       " " port " " worker-id)]      (log-message "Launching worker with command: " command)      (launch-process command :environment {"LD_LIBRARY_PATH" (conf JAVA-LIBRARY-PATH)})      ))

以上便是strom的几种日志生成方式。

在storm的开发中，我们通常会自定义日志路径，方便管理

如定义路径为${storm.home}/logs/${topo_name}/${logfile.name}

0 0