nginx日志解析输出elasticstack

nginx解析elastic stack(采集，解析分离)

需求：实时读取nginx access日志经过采集，缓存，最终导入到elasticsearch平台进行展示查询。由于nginx日志源主机与elasticsearch平台在不同的内网网段内，所以没有采用通常直接的logstash读取并解析然后直接输出到elasticsearch。而是在日志源运行logstash作为采集并转发到隔离网关的redis集群，然后内网内部运行logstash从redis读取并解析再输出到elasticsearch平台。

采集

日志采集的logstash角色被称为shipping，仅读取并转存，不进行格式解析，使用如下配置:

input {    file {        path => "/var/log/nginx/access.log"        start_position => beginning        codec => plain {            charset => "UTF-8"        }        type => "nginx"    }}output {    redis {        host => "192.168.100.42"        port => 7021        data_type => "list"            key => "logstash-key"    }    stdout {        codec => rubydebug    }}#注： redis-cli连接redis，查看list类型的key值lpop "logstash-key"    

---

解析

解析端的logstash从redis集群读取list，并按照nginx日志模板解析输出到elasticsearch平台。

input {    redis {        host => "192.168.100.42"        port => 7021        type => "redis_input"        codec => "json"        data_type => "list"        key => "logstash-key"    #该key与采集端的logstash:redis key一致    }}filter {    grok {        patterns_dir => "/usr/local/logstash-5.1.1/patt"        match => {            "message" => "%{NGINXACCESS}%"        }        overwrite => ["message"]    }    date {        match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]    }}output {    elasticsearch {        hosts => "192.168.100.41:9200"        index => "shopweb_nginx"    }    stdout {        codec => rubydebug    }}

## grok模板CACHE_STAT \w+|-RE_TIME %{NUMBER}|-the_URI %{URI}|-NGINXACCESS1 %{IP:client} - - \[%{HTTPDATE:localtime}\] \"%{WORD:method} %{URIPATHPARAM:uri_parm} HTTP/%{NUMBER:ver}\" %{NUMBER:status:int} %{NUMBER:body_bytes_sent:int} \"%{the_URI:referer}\" %{NUMBER:bytes_sent:int} %{NUMBER:request_length:int} \"%{GREEDYDATA:agent}\" \"-\" \"%{CACHE_STAT:cache_status}\" %{RE_TIME:request_time} %{RE_TIME:up_response_time}