logstash使用小结
来源：互联网 发布：博客平台推荐 知乎 编辑：程序博客网 时间：2024/05/20 17:38
http://nosmoking.blog.51cto.com/3263888/1852115
一、安装1、jdk 和 环境变量支持jdk-1.7以上，推荐jdk-1.8在环境变量配置：JAVA_HOME2、安装有2种方式下载，推荐缓存rpm包到本地yum源1）直接使用rpmwget https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.4.0.noarch.rpm2）使用yum源[root@vm49 ~]# rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch[root@vm49 ~]# vim /etc/yum.repos.d/logstash.repo[logstash-2.4]name=Logstash repository for 2.4.x packagesbaseurl=https://packages.elastic.co/logstash/2.4/centosgpgcheck=1gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearchenabled=1[root@vm49 ~]# yum install logstash[root@vm49 ~]# whereis logstashlogstash: /etc/logstash /opt/logstash/bin/logstash /opt/logstash/bin/logstash.bat二、使用1、命令行测试1）输出到 stdout[root@vm49 ~]# /opt/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'hi, let us go（输入）Settings: Default pipeline workers: 4Pipeline main started2016-09-12T02:42:59.110Z 0.0.0.0 hi, let us go（输出）why not TRY IT OUT（输入）2016-09-12T02:43:11.904Z 0.0.0.0 why not TRY IT OUT（输出）（CTRL-D 退出）Pipeline main has been shutdownstopping pipeline {:id=>"main"}2）改变输出到 stdout 的格式[root@vm49 ~]# /opt/logstash/bin/logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }'aaaaSettings: Default pipeline workers: 4Pipeline main started{ "message" => "aaaa", "@version" => "1", "@timestamp" => "2016-09-18T02:11:07.109Z", "host" => "0.0.0.0"}bbbb{ "message" => "bbbb", "@version" => "1", "@timestamp" => "2016-09-18T02:11:11.864Z", "host" => "0.0.0.0"}Pipeline main has been shutdownstopping pipeline {:id=>"main"}3）上述输出发现一个小问题，得到的 host 为 0.0.0.0，而不是期望中的当前 hostname检查一下 hostname[root@vm49 ~]# hostnamevm49.localdomain[root@vm49 ~]# echo '10.50.200.49 vm49.localdomain' >>/etc/hosts再次测试：[root@vm49 ~]# /opt/logstash/bin/logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }'aaaaaSettings: Default pipeline workers: 4Pipeline main started{ "message" => "aaaaa", "@version" => "1", "@timestamp" => "2016-09-18T02:22:12.180Z", "host" => "vm49.localdomain"}Pipeline main has been shutdownstopping pipeline {:id=>"main"}符合预期。注1：还有一种场景（hostname使用本地名称，而不是采取域名的命名方式）需要注意/etc/hosts 中包括下述内容------------------------127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6127.0.0.1 local-vm5010.50.200.50 local-vm50------------------------期望的结果是："host" => "local-vm50"实际得到的结果是："host" => "localhost"此时，需要删除：127.0.0.1 local-vm50然后才能得到期望的结果。注2：如果是在 logstash 服务启动后才去调整了 hostname 相关设置，则需要重启 logstash 服务，否则依然是旧的结果。2、使用配置文件目的：从日志文件中读取数据，输出到另一个文件中来查看。前提：已经配置了一个nginx服务，生成了对应的日志文件。首先，因为这里要读取nginx日志，请将 logstash 加入 nginx 的用户组。示例：[root@vm49 ~]# ls -l /var/log/nginx/access_*.log-rw-r----- 1 nginx adm 608083 Sep 13 18:18 /var/log/nginx/access_www.test.com_80.log-rw-r----- 1 nginx adm 597070 Sep 13 18:18 /var/log/nginx/access_www.work.com_80.lognginx属于 adm 组，使用 logstash 读取日志，可能产生权限异常：failed to open /var/log/nginx/access.ecshop.test.d3dstore.com.log: Permission denied因此，要加入对应的组：[root@vm49 ~]# usermod -G adm logstash接着，我们尝试这样配置 logstash 来收集日志：[root@vm49 ~]# cat /etc/logstash/conf.d/nginx.conf input { file { path => "/var/log/nginx/access_*.log" start_position => beginning ignore_older => 0 }}filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}"} }}output { file { path => "/tmp/test.log" }}上面使用到以下插件：file：日志数据的输入和输出grok：来匹配标准的apache日志格式【细节延伸】显然，在3个环节中，都有改进和调整的地方。input：使用 filebeatfilter：使用其他插件和规则output：使用ES，redis等具体请参考：https://www.elastic.co/guide/en/logstash/current/pipeline.html3、测试配置文件：[root@vm49 ~]# service logstash configtestConfiguration OK4、启动服务：[root@vm49 ~]# service logstash start[root@vm49 ~]# chkconfig logstash on5、测试请求nginx服务，然后观察输出的内容：[root@vm49 ~]# cat /tmp/test.log符合预期。6、比较去掉 filter 这一节，我们来对比一下 /tmp/test.log 收集到的内容的差异【使用了 filter 的结果a】{"message":"10.50.200.219 - - [12/Sep/2016:13:00:03 +0800] \"GET / HTTP/1.1\" 200 13 \"-\" \"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2\" \"-\" 0.000 \"-\" \"-\"","@version":"1","@timestamp":"2016-09-12T05:00:04.140Z","path":"/var/log/nginx/access_www.test.com_80.log","host":"vm49.localdomain","clientip":"10.50.200.219","ident":"-","auth":"-","timestamp":"12/Sep/2016:13:00:03 +0800","verb":"GET","request":"/","httpversion":"1.1","response":"200","bytes":"13","referrer":"\"-\"","agent":"\"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2\""}【未使用 filter 的结果b】{"message":"10.50.200.219 - - [12/Sep/2016:13:07:49 +0800] \"GET / HTTP/1.1\" 200 13 \"-\" \"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2\" \"-\" 0.000 \"-\" \"-\"","@version":"1","@timestamp":"2016-09-12T05:07:49.917Z","path":"/var/log/nginx/access_www.test.com_80.log","host":"vm49.localdomain"}a的内容中，多出来的地方，正是使用了 grok 分析和结构化数据---------------------------------------------------Information Field Name----------- ----------IP Address clientipUser ID identUser Authentication authtimestamp timestampHTTP Verb verbRequest body requestHTTP Version httpversionHTTP Status Code responseBytes served bytesReferrer URL referrerUser agent agent---------------------------------------------------注：也可以将 filter 调整到服务端，让客户端仅仅是收集日志即可，这个取舍需要自行对比。7、改进Logstash 默认自带了 apache 标准日志的 grok 正则:如何使用自定义的日志格式呢？例如，默认的 nginx 日志是： log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';改成自定义的日志格式： log_format online '$remote_addr [$time_local] "$request" ' '"$http_content_type" "$request_body" "$http_referer" ' '$status $request_time $body_bytes_sent'; 对应的数据：【GET】# curl -H "Content-Type: text/html; charset=UTF-8" --referer 'www.abc.com/this_is_a_referer' http://www.test.com/a/b/c.html?key1=value1【结果】10.50.200.219 [12/Sep/2016:15:11:04 +0800] "GET /a/b/c.html?key1=value1 HTTP/1.1" "text/html; charset=UTF-8" "-" "www.abc.com/this_is_a_referer" 404 0.000 168【POST】# curl -H "Content-Type: application/xml" -d "{"name": "Mark Lee" }" "http://www.test.com/start"【结果】10.50.200.218 [12/Sep/2016:15:02:07 +0800] "POST /start HTTP/1.1" "application/xml" "-" "-" 404 0.000 168尝试一下：[root@vm49 ~]# mkdir -p /etc/logstash/patterns.d[root@vm49 ~]# vim /etc/logstash/patterns.d/extra_patternsNGINXACCESS %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" (?:%{QS:content_type}|-) (?:%{QS:request_body}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{NUMBER:response} %{BASE16FLOAT:request_time} (?:%{NUMBER:bytes}|-)调整配置为：[root@vm49 ~]# cat /etc/logstash/conf.d/nginx.conf input { file { path => "/var/log/nginx/access_*.log" start_position => beginning ignore_older => 0 }}filter { grok { patterns_dir => ["/etc/logstash/patterns.d"] match => { "message" => "%{NGINXACCESS}" } }}output { file { path => "/tmp/test.log" }}[root@vm49 ~]# service logstash restart结果：{"message":"10.50.200.218 [12/Sep/2016:15:28:23 +0800] \"POST /start HTTP/1.1\" \"application/xml\" \"-\" \"-\" 404 0.000 168","@version":"1","@timestamp":"2016-09-12T07:28:24.007Z","path":"/var/log/nginx/access_www.test.com_80.log","host":"vm49.localdomain","clientip":"10.50.200.218","timestamp":"12/Sep/2016:15:28:23 +0800","verb":"POST","request":"/start","httpversion":"1.1","content_type":"\"application/xml\"","request_body":"\"-\"","response":"404","request_time":"0.000","bytes":"168"}{"message":"10.50.200.219 [12/Sep/2016:15:28:24 +0800] \"GET /a/b/c.html?key1=value1 HTTP/1.1\" \"text/html; charset=UTF-8\" \"-\" \"www.abc.com/this_is_a_referer\" 404 0.000 168","@version":"1","@timestamp":"2016-09-12T07:28:25.019Z","path":"/var/log/nginx/access_www.test.com_80.log","host":"vm49.localdomain","clientip":"10.50.200.219","timestamp":"12/Sep/2016:15:28:24 +0800","verb":"GET","request":"/a/b/c.html?key1=value1","httpversion":"1.1","content_type":"\"text/html; charset=UTF-8\"","request_body":"\"-\"","referrer":"\"www.abc.com/this_is_a_referer\"","response":"404","request_time":"0.000","bytes":"168"}符合预期。三、输出到 redis+elasticsearch+kibana1、测试环境（已经部署了服务）【客户端】10.50.200.49: logstash, nginx(www.test.com, www.work.com)【服务端】10.50.200.220: logstash, redis, elasticsearch, kibana【测试端】10.50.200.218, 10.50.200.219: curl 请求 nginx1）测试脚本--------------------------------------------------------[root@vm218 bin]# cat test_www_by_curl.sh#!/bin/bash##2016/9/20# crontab:# */2 * * * * /usr/local/bin/test_www_by_curl.sh >>/tmp/test_www_by_curl.log 2>&1 &echo "[S] at: `date`"tag="$(echo `hostname` |awk -F'-' '{print $NF}')"do_curl(){ www=$1 rnd=$2 #PUT curl -H "Content-Type: application/xml" -d "{"name": "Jack A"}" "http://${www}/from_${tag}" curl -H "Content-Type: application/json" -d "{"name": "Jack B"}" "http://${www}/from_${tag}" curl -H "Content-Type: application/png" -d "{"img_path": "images/$(date +%F)/${rnd}.png"}" "http://${www}/from_${tag}" #GET curl -H "Content-Type: text/html; charset=UTF-8" --referer "www.${tag}.com/this_is_referer" "http://${www}/" curl -H "Content-Type: text/html; charset=UTF-8" --referer "www.${tag}.com/this_is_referer" "http://${www}/a/b/c.html?key=${tag}" curl -H "Content-Type: text/html; charset=GBK" --referer "www.${tag}.com/this_is_referer" "http://${www}/action.do?event=login&user=${tag}&sid=${rnd}" #}for i in `seq 1 2000`; do do_curl www.work.com "`od /dev/urandom -w12 -tx4 -An |sed -e 's/ //g' |head -n 1`" do_curl www.test.com "`od /dev/urandom -w12 -tx4 -An |sed -e 's/ //g' |head -n 1`"done >/dev/null 2>&1echo "[D] at: `date`"--------------------------------------------------------2）hosts 写入域名10.50.200.49 www.test.com10.50.200.49 www.work.com2、场景1：只有1个域名/模糊匹配N个域名目的：将匹配的 access 日志收集起来集中展示。【客户端】输入：file输出：redis[root@vm49 ~]# cat /etc/logstash/conf.d/nginx.conf input { file { type => "NginxAccess" path => "/var/log/nginx/access_*.log" start_position => beginning ignore_older => 0 }}filter { if[type] == "NginxAccess" { grok { patterns_dir => ["/etc/logstash/patterns.d"] match => { "message" => "%{NGINXACCESS}" } } date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] remove_field => [ "timestamp" ] } }}output { if[type] == "NginxAccess" { redis { host => "10.50.200.220" data_type => "list" key => "logstash:nginxaccess" } }}[root@vm49 ~]# service logstash restart【服务端】输入：redis输出：elasticsearch[root@vm220 ~]# vim /etc/logstash/conf.d/redis.confinput { redis { host => '127.0.0.1' data_type => 'list' port => "6379" key => 'logstash:nginxaccess' type => 'redis-input' }}output { if[type] == "NginxAccess" { elasticsearch { hosts => "127.0.0.1:9200" index => "logstash-nginxaccess-%{+YYYY.MM.dd}" } }}[root@vm220 ~]# service logstash restart可以通过命令行去观察 redis 的状态：[root@vm220 ~]# redis-cli monitor指定kibana的索引名称，例如：logstash-nginxaccess-*结果：符合预期。3、场景2：N个域名分开收集目的：将 www.test.com 和 www.work.com 的 access 日志收集起来使用独立的索引名来分开展示【客户端】输入：file输出：redis[root@vm49 ~]# cat /etc/logstash/conf.d/nginx.conf input { file { type => "NginxAccess-www.test.com" path => "/var/log/nginx/access_www.test.com*.log" start_position => beginning ignore_older => 0 } file { type => "NginxAccess-www.work.com" path => "/var/log/nginx/access_www.work.com*.log" start_position => beginning ignore_older => 0 }}filter { if[type] =~ "NginxAccess-" { grok { patterns_dir => ["/etc/logstash/patterns.d"] match => { "message" => "%{NGINXACCESS}" } } date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] remove_field => [ "timestamp" ] } }}output { if[type] =~ "NginxAccess-" { redis { host => "10.50.200.220" data_type => "list" key => "logstash:nginxaccess" } }}[root@vm49 ~]# service logstash restart【服务端】输入：redis输出：elasticsearch[root@vm220 ~]# cat /etc/logstash/conf.d/redis.conf input { redis { host => '127.0.0.1' data_type => 'list' port => "6379" key => 'logstash:nginxaccess' type => 'redis-input' }}output { if[type] == "NginxAccess-www.test.com" { elasticsearch { hosts => "127.0.0.1:9200" index => "logstash-nginxaccess-www.test.com-%{+YYYY.MM.dd}" } } if[type] == "NginxAccess-www.work.com" { elasticsearch { hosts => "127.0.0.1:9200" index => "logstash-nginxaccess-www.work.com-%{+YYYY.MM.dd}" } }}[root@vm220 ~]# service logstash restart当然了，要调整kibana的索引名称，指定更具体的名称，例如：logstash-nginxaccess-www.test.com-*logstash-nginxaccess-www.work.com-*结果：符合预期。四、小结FAQ1、数据流向------------------------------------------------------------------log_files -> logstash -> redis -> elasticsearch -> kibana------------------------------------------------------------------2、@timestamp 和 message 中记录的时间字段（timestamp）不一致message 中记录的是原始时间，而 @timestamp 中记录的是事件采集到的时候记录的时间，两者可能差距有1s及以上。可以通过 date 插件来调整记录到的时间，以日志中记录的原始时间为准，可以用于读取过去某段时间的日志文件，这样一来，收集日志时记录的时间不会错乱。例如：filter { date { match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ] remove_field => [ "timestamp" ] }}将匹配 field "timestamp" 格式为 "dd/MMM/YYYY:HH:mm:ss Z"实际匹配到：18/Sep/2016:16:43:15 +0800默认更新 @timestamp 为：September 18th 2016, 16:43:15.000特别说明：1）timestamp 是我们在原始日志中收集的时间字段，不一定是这个名称（这意味是任意的自定义的字段名称），因为只是我们在前述操作中使用 grok 插件中在 patterns 中定义了 "%{NGINXACCESS}" 这个变量来解析原始字段，里边定义了这个 "timestamp" 字段。a、grok插件的定义 grok { patterns_dir => ["/etc/logstash/patterns.d"] match => { "message" => "%{NGINXACCESS}" } }b、grok插件指明的 patterns 的定义[root@vm49 ~]# cat /etc/logstash/patterns.d/extra_patternsNGINXACCESS %{IPORHOST:clientip} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" (?:%{QS:content_type}|-) (?:%{QS:request_body}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{NUMBER:response} %{BASE16FLOAT:request_time} (?:%{NUMBER:bytes}|-)2）为何要使用 remove_field 功能？a、重复数据我们已经有了一个时间字段 "@timestamp" 因此可以丢掉这个重复的 "timestamp" 字段。注：由此联想一下，生产应用中，我们是不是应该好好利用下 "remove_field" 功能去筛选一下字段呢？主动的去丢弃一些字段，只保留特别重要的指标发送给 output 即可。b、导入到 elasticsearch 时，模版 mapping type 时异常以收集 nginx error log 为例：error log 的时间字段格式是这样的：2016/09/21 10:34:41对应的 date 插件是这样写的： date { match => [ "timestamp", "YYYY/MM/dd HH:mm:ss" ] remove_field => [ "timestamp" ] }如果不删除字段 "timestamp"，当传递到 elasticsearch 时，模版将试图解析这字段的格式，这个时候会遇到解析异常，从而影响到 index 的建立。在 elasticsearch 的日志中将产生如下错误：MapperParsingException[failed to parse [timestamp]]; nested: IllegalArgumentException[Invalid format: "2016/09/21 10:34:41" is malformed at "/09/21 10:34:41"];3、如何使用 grok 插件来匹配 nginx error log1）示例日志2016/09/21 10:34:41 [error] 11990#0: *6718852 no live upstreams while connecting to upstream, client: 10.50.200.219, server: c.test.com, request: "GET http://www.work.com/cgi-bin/common/attr?id=260714&r=0.1011513018987016 HTTP/1.1", upstream: "http://upstream_backend/cgi-bin/common/attr?id=260714&r=0.1011513018987016", host: "www.work.com"2016/09/21 11:26:41 [error] 1614#0: *25584123 open() "/data/www/www.test.com/404.html" failed (2: No such file or directory), client: 10.50.200.218, server: www.test.com, request: "GET /action.do?event=login&user=vm218&sid=15120ae826ca2290514bcd5b HTTP/1.1", host: "www.test.com", referrer: "www.vm218.com/this_is_referer"2）示例匹配规则NGINXERROR_DATESTAMP %{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME}NGINXERROR_PID (?:[0-9]+#[0-9]+\:)NGINXERROR_TID (?:\*[0-9]+)NGINXERROR %{NGINXERROR_DATESTAMP:timestamp} \[%{LOGLEVEL:loglevel}\] %{NGINXERROR_PID:pid} %{NGINXERROR_TID:tid} %{GREEDYDATA:errormsg}, client: %{IPORHOST:clientip}, server: %{HOSTNAME:server}, request: %{QS:request}(?:, upstream: %{QS:upstream})?, host: \"%{HOSTNAME:hostname}\"(?:, referrer: (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}))?注：特别说明，“%{NGINXERROR_PID:pid} %{NGINXERROR_TID:tid}” 这一段只是为了匹配“11990#0: *6718852”并打算丢弃掉，并非实际需要这2个字段。3）使用工具测试http://grokconstructor.appspot.com/do/match4）结果------------------------------------------------------------------2016/09/21 10:34:41 [error] 11990#0: *6718852 no live upstreams while connecting to upstream, client: 10.50.200.219, server: c.test.com, request: "GET http://www.work.com/cgi-bin/common/attr?id=260714&r=0.1011513018987016 HTTP/1.1", upstream: "http://upstream_backend/cgi-bin/common/attr?id=260714&r=0.1011513018987016", host: "www.work.com"【MATCHED】request "GET·http://www.work.com/cgi-bin/common/attr?id=260714&r=0.1011513018987016·HTTP/1.1"server c.test.comhostname www.work.comtimestamp 2016/09/21·10:34:41clientip 10.50.200.219port pid 11990#0:upstream "http://upstream_backend/cgi-bin/common/attr?id=260714&r=0.1011513018987016"referrer tid *6718852errormsg no·live·upstreams·while·connecting·to·upstreamloglevel error==================================================================2016/09/21 11:26:41 [error] 1614#0: *25584123 open() "/data/www/www.test.com/404.html" failed (2: No such file or directory), client: 10.50.200.218, server: www.test.com, request: "GET /action.do?event=login&user=vm218&sid=15120ae826ca2290514bcd5b HTTP/1.1", host: "www.test.com", referrer: "www.vm218.com/this_is_referer"【MATCHED】request "GET·/action.do?event=login&user=vm218&sid=15120ae826ca2290514bcd5b·HTTP/1.1"server www.test.comhostname www.test.comtimestamp 2016/09/21·11:26:41clientip 10.50.200.218port pid 1614#0:upstream referrer "www.vm218.com/this_is_referer"tid *25584123errormsg open()·"/data/www/www.test.com/404.html"·failed·(2:·No·such·file·or·directory)loglevel error ------------------------------------------------------------------5）应用【服务端】a、增加上述匹配规则到配置文件[root@vm220 ~]# cat /etc/logstash/patterns.d/extra_patternsNGINXERROR_DATESTAMP %{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME}NGINXERROR_PID (?:[0-9]+#[0-9]+\:)NGINXERROR_TID (?:\*[0-9]+)NGINXERROR %{NGINXERROR_DATESTAMP:timestamp} \[%{LOGLEVEL:loglevel}\] %{NGINXERROR_PID:pid} %{NGINXERROR_TID:tid} %{GREEDYDATA:errormsg}, client: %{IPORHOST:clientip}, server: %{HOSTNAME:server}, request: %{QS:request}(?:, upstream: %{QS:upstream})?, host: \"%{HOSTNAME:hostname}\"(?:, referrer: (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}))?b、调整对应的 logstash 配置[root@vm220 ~]# vim /etc/logstash/conf.d/filebeat.conf 在 filter 这一段增加： if[type] =~ "NginxError-" { grok { patterns_dir => ["/etc/logstash/patterns.d"] match => { "message" => "%{NGINXERROR}" } } date { match => [ "timestamp", "YYYY/MM/dd HH:mm:ss" ] remove_field => [ "timestamp" ] } }在 output 这一段增加： if[type] == "NginxError-www.test.com" { elasticsearch { hosts => ["10.50.200.218:9200", "10.50.200.219:9200", "10.50.200.220:9200"] manage_template => false index => "%{[@metadata][beat]}-nginxerror-www.test.com-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } if[type] == "NginxError-www.work.com" { elasticsearch { hosts => ["10.50.200.218:9200", "10.50.200.219:9200", "10.50.200.220:9200"] manage_template => false index => "%{[@metadata][beat]}-nginxerror-www.work.com-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } }c、重启 logstash 服务[root@vm220 ~]# service logstash restart【客户端】d、调整对应的 filebeat 配置[root@vm49 ~]# vim /etc/filebeat/filebeat.yml（增加对应的 prospectors 来收集日志，打上 type 标记，其他配置略） prospectors: - paths: - /var/log/nginx/error_www.test.com*.log input_type: log document_type: NginxError-www.test.com - paths: - /var/log/nginx/error_www.work.com*.log input_type: log document_type: NginxError-www.work.come、重启 filebeat 服务[root@vm49 ~]# service filebeat restart4、怎么调试问题直接改变 output 到 stdout 即可查看输出是否符合需求。output { if[type] =~ "NginxAccess-" { stdout { codec => rubydebug } } if[type] =~ "NginxError-" { stdout { codec => rubydebug } }}ZYXW、参考1、官网https://www.elastic.co/guide/en/logstash/current/introduction.htmlhttps://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.htmlhttps://www.elastic.co/guide/en/logstash/current/installing-logstash.htmlhttps://www.elastic.co/guide/en/logstash/current/first-event.htmlhttps://www.elastic.co/guide/en/logstash/current/advanced-pipeline.htmlhttps://www.elastic.co/guide/en/logstash/current/pipeline.htmlhttps://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.htmlhttps://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html2、ELK中文http://kibana.logstash.es/content/http://kibana.logstash.es/content/logstash/plugins/filter/grok.htmlhttp://kibana.logstash.es/content/beats/file.html3、用ELK搭建简单的日志收集分析系统http://blog.csdn.net/lzw_2006/article/details/512800584、logstash host field is 0.0.0.0http://www.v1en.com/2016/03/31/logstash-host-field-is-0-0-0-0/5、grokhttps://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patternshttp://grokconstructor.appspot.com/do/match 0 0
- logstash使用小结
- logstash安装和使用
- 开始使用Logstash
- logstash 配置使用
- Logstash使用详解
- logstash安装与使用
- Logstash 使用详解
- Logstash安装使用
- Logstash的使用
- logstash使用之简单使用
- 使用logstash分析Apache日志
- Logstash的安装和使用
- LogStash的Filter的使用
- logstash-output elasticsearch插件使用
- logstash-input-exec插件使用
- logstash的kafka插件使用
- ELK中logstash的使用
- Logstash的kafka插件使用
- 设计模式之---迭代器模式
- Tensorflow学习：Variable变量
- jmeter使用put方法提交的数据格式
- 使用Connetion的属性RetainSameConnection
- 后缀数组总结
- logstash使用小结
- LEADTOOLS Recognition Imaging Developer Toolkit
- java处理日期时间
- Spring mvc中的@RequestHeader
- Java学习之HashSet
- input子系统一 主要数据结构
- POI操作Excel的简单操作
- Linux 程序包
- 《APUE》读书笔记-第二十章数据库函数库
