Logstash -- Nginx 日志收集处理!

Nginx 日志收集示例一：

Logstash 默认自带了 apache 标准日志的 grok 正则:

COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{NOTSPACE:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

对于 nginx 标准日志格式，可以发现只是最后多了一个 $http_x_forwarded_for 变量。所以 nginx 标准日志的 grok 正则定义是：

MAINNGINXLOG %{COMBINEDAPACHELOG} %{QS:x_forwarded_for}

• logstash 配置文件01[直接使用nginx默认的logformat]：

[root@ ~]# cat /logstash/config/nginx-test01.confinput {    file {        path => "/tmp/nginx/*access*.log"        start_position => beginning    }}filter {    grok {        match => { "message" => "%{COMBINEDAPACHELOG} %{QS:x_forwarded_for}"}    }    date {        match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]    }    geoip {        source => "clientip"    }}output {    elasticsearch {     hosts => "192.168.11.10:9200"     }    stdout { codec => rubydebug }}# ./bin/logstash -f config/nginx-test01.conf

Nginx 日志收集示例二：

• 需要对Nginx的配置文件中的日志格式进行定制

# cat /etc/nginx/nginx.conf............http {    include       /etc/nginx/mime.types;    default_type  application/octet-stream;    upstream elasticsearch {           server 192.168.11.10:9200;           keepalive 15;       }    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '    #                  '$status $body_bytes_sent "$http_referer" '    #                  '"$http_user_agent" "$http_x_forwarded_for"';    log_format    json '{"@timestamp":"$time_iso8601",                      "@version":"1","host":"$server_addr",                      "client":"$remote_addr", "size":"$body_bytes_sent,                      "responsetime":"$request_time",                      "domain":"$host","url":"$uri","status":"$status"}'    access_log  /var/log/access_json.log  json;    sendfile        on;    #tcp_nopush     on;............ 省略# cat /opt/logstash/config/nginx-test02.conf input {    file {        path => "/var/log/nginx/*access*"        codec => "json"    }}output {  elasticsearch { hosts => ["192.168.11.10:9200"] }  stdout { codec => rubydebug }}# ./bin/logstash -f config/nginx-test02.conf

在实际应用中 选取其中一种方式收集即可！

最后查看kibana展示出来的信息即可！

---

参考链接

elastic 权威指南
logstash 官方示例