Redhat linux 下 fluentd+fluentd 实现桥接

实验环境为2台测试机分别为Server201（192.168.10.201）和Server202（192.168.19.202），分别安装好fluentd，具体安装步骤不再累述。

重点描述2台机器上的配置，目的是将Server202上的日志，通过fluentd桥接的方式，放置到Server201下的某个目录。

1、Server202上fluentd.conf：

<source>
 type tail 

#需要转储的日志1
 path /home/aircom/mongodb-linux-i686-2.0.6/bin/logs/1001.log
format /^*(?<message>.*)$/
 tag mongo.apache
</source>
<source>
 type tail

#需要转储的日志2
 path /home/aircom/mongodb-linux-i686-2.0.6/bin/logs/1002.log
format /^*(?<message>.*)$/
 tag mongo.apache
</source>

源日志为2个日志文件。分别为path /home/aircom/mongodb-linux-i686-2.0.6/bin/logs/1001.log和path /home/aircom/mongodb-linux-i686-2.0.6/bin/logs/1002.log

标签为tag mongo.apache，方便match段匹配。

<match mongo.**>
  type forward

#转储服务器
  host 192.168.19.201
  <secondary>
    host 192.168.19.201
  </secondary>
</match>

分别表示了2台机器，可以添加备用机，这里的主、备机IP均相同，生产环境可以添加不同的IP。

2、Server201上fluent.conf配置

为了方便测试，我们直接修改，将注释直接打开，其含义为匹配没有match的日志，写入到文件。

<match **>
  type file

#存储路径
  path /var/log/fluent/else
  compress gz
</match>

启动fluentd

201上查看配置的路径

[root@Server201 fluentd-0.10.6]# ll  /var/log/fluent/
total 68
-rw-r--r-- 1 root root 61675 Sep 28 10:20 else.20120928.b4cab973e8d931ee9

产生了文件else.20120928.b4cab973e8d931ee9

3、启动Server202下的fluentd：fluentd -c fluent.conf -deamon -o log/log.log

tail 输入日志，得到后几行输出

2012-09-28 09:58:45 +0800: adding match pattern="mongo.**" type="forward"
2012-09-28 09:58:45 +0800: 'host' option in forward output is obsoleted. Use '<server> host xxx </server>' instead.
2012-09-28 09:58:45 +0800: adding forwarding server '192.168.19.201:24224' host="192.168.19.201" port=24224 weight=60
2012-09-28 09:58:45 +0800: 'host' option in forward output is obsoleted. Use '<server> host xxx </server>' instead.
2012-09-28 09:58:45 +0800: adding forwarding server '192.168.19.201:24224' host="192.168.19.201" port=24224 weight=60
2012-09-28 09:58:45 +0800: listening fluent socket on 0.0.0.0:24224
2012-09-28 09:58:45 +0800: following tail of /home/aircom/mongodb-linux-i686-2.0.6/bin/logs/1001.log
2012-09-28 09:58:45 +0800: following tail of /home/aircom/mongodb-linux-i686-2.0.6/bin/logs/1002.log

标记颜色的2行表示2台接受数据的机子，端口为24224，

开始发送数据的时候这个日志还会得到如下输出

2012-09-28 10:02:07 +0800: detached forwarding server '192.168.19.201:24224' host="192.168.19.201" port=24224 phi=8.280932634573869
2012-09-28 10:02:07 +0800: detached forwarding server '192.168.19.201:24224' host="192.168.19.201" port=24224 phi=8.280785193737733
2012-09-28 10:02:42 +0800: recovered forwarding server '192.168.19.201:24224' host="192.168.19.201" port=24224
2012-09-28 10:02:42 +0800: recovered forwarding server '192.168.19.201:24224' host="192.168.19.201" port=24224

4、查看Server201上桥接转储后的日志

tail -f /var/log/fluent/else.20120928.b4cab973e8d931ee9

大概显示如下输出，日志刷新频次为1分钟

2012-09-28 10:23:09 +0800: plugin/in_forward.rb:140:initialize: accepted fluent socket object_id=242397940
2012-09-28 10:23:09 +0800: plugin/in_forward.rb:181:on_close: closed fluent socket object_id=242397940
2012-09-28T10:23:09+08:00       fluent.trace    {"message":"accepted fluent socket object_id=242397940"}
2012-09-28T10:22:23+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:23 [conn369] end connection 192.168.19.201:51962"}
2012-09-28T10:22:23+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:23 [initandlisten] connection accepted from 192.168.19.201:51968 #372"}
2012-09-28T10:22:27+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:27 [conn370] end connection 192.168.19.201:51963"}
2012-09-28T10:22:27+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:27 [initandlisten] connection accepted from 192.168.19.201:51969 #373"}
2012-09-28T10:22:33+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:33 [conn371] end connection 192.168.19.202:39038"}
2012-09-28T10:22:33+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:33 [initandlisten] connection accepted from 192.168.19.202:39045 #374"}
2012-09-28T10:22:34+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:34 [conn378] end connection 192.168.19.202:55169"}
2012-09-28T10:22:34+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:34 [initandlisten] connection accepted from 192.168.19.202:55176 #381"}
2012-09-28T10:22:35+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:35 [conn379] end connection 192.168.19.201:41497"}
2012-09-28T10:22:35+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:35 [initandlisten] connection accepted from 192.168.19.201:41503 #382"}
2012-09-28T10:22:37+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:37 [conn380] end connection 192.168.19.201:41498"}
2012-09-28T10:22:37+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:37 [initandlisten] connection accepted from 192.168.19.201:41504 #383"}
2012-09-28T10:22:53+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:53 [conn372] end connection 192.168.19.201:51968"}
2012-09-28T10:22:53+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:53 [initandlisten] connection accepted from 192.168.19.201:51974 #375"}
2012-09-28T10:22:56+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:56 [clientcursormon] mem (MB) res:76 virt:419 mapped:224"}
2012-09-28T10:22:57+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:57 [conn373] end connection 192.168.19.201:51969"}
2012-09-28T10:22:57+08:00       mongo.apache    {"message":"Fri Sep 28 10:22:57 [initandlisten] connection accepted from 192.168.19.201:51975 #376"}
2012-09-28T10:23:01+08:00       mongo.apache    {"message":"Fri Sep 28 10:23:01 [clientcursormon] mem (MB) res:44 virt:329 mapped:160"}
2012-09-28T10:23:03+08:00       mongo.apache    {"message":"Fri Sep 28 10:23:03 [conn374] end connection 192.168.19.202:39045"}
2012-09-28T10:23:03+08:00       mongo.apache    {"message":"Fri Sep 28 10:23:03 [initandlisten] connection accepted from 192.168.19.202:39051 #377"}
2012-09-28T10:23:04+08:00       mongo.apache    {"message":"Fri Sep 28 10:23:04 [conn381] end connection 192.168.19.202:55176"}
2012-09-28T10:23:04+08:00       mongo.apache    {"message":"Fri Sep 28 10:23:04 [initandlisten] connection accepted from 192.168.19.202:55182 #384"}
2012-09-28T10:23:05+08:00       mongo.apache    {"message":"Fri Sep 28 10:23:05 [conn382] end connection 192.168.19.201:41503"}
2012-09-28T10:23:05+08:00       mongo.apache    {"message":"Fri Sep 28 10:23:05 [initandlisten] connection accepted from 192.168.19.201:41509 #385"}
2012-09-28T10:23:07+08:00       mongo.apache    {"message":"Fri Sep 28 10:23:07 [conn383] end connection 192.168.19.201:41504"}
2012-09-28T10:23:07+08:00       mongo.apache    {"message":"Fri Sep 28 10:23:07 [initandlisten] connection accepted from 192.168.19.201:41510 #386"}
2012-09-28T10:23:09+08:00       fluent.trace    {"message":"closed fluent socket object_id=242397940"}

至此已经完全实现2台服务器上2个fluentd的桥接。