Logstash+Redis+Elasticsearch+Kibana 快速搭建Nginx日志查询系统

来源：互联网发布：java里定义全局变量编辑：程序博客网时间：2024/05/21 19:41

Logstash+Redis+Elasticsearch+Kibana Nginx日志查询系统

环境

elasticsearch-0.90.5.zipkibana-latest.zipredis-2.6.16.tar.gzlogstash-1.2.2-flatjar.jar

nginx.conf配置

log_format  main  '$remote_addr - $remote_user [$time_local] '                            '"$request" $status $body_bytes_sent '                           '"$http_referer" "$http_user_agent" ';

nginx日志

172.16.201.174 - - [25/Mar/2014:16:39:13 +0800] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1772.0 Safari/537.36"

表达式

%{IPORHOST:source_ip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}

collection

hadoop@stormspark:~/log/logstash$ cat  sp.conf input {   file {       type => "nginx-access"       path => "/var/log/nginx/access.log"   }}output {    stdout {        debug => true        debug_format => json    }    redis {        host => "127.0.0.1"        port => 6379        data_type => "list"        key => "logstash"    }}

index配置

hadoop@stormspark:~/log/logstash$ cat index.conf input {  redis {    host => "127.0.0.1"    port => "6379"    data_type => "list"    key => "logstash"    type => "redis-input"  }}filter {   grok {       type => "nginx-access"       pattern => "%{IPORHOST:source_ip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] %{QS:request} %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}"   }}output {  elasticsearch {    host => "127.0.0.1"  }}

分别启动logstash,redis,es等。

java -jar logstash-1.2.2-flatjar.jar agent -f sp.confjava -jar logstash-1.2.2-flatjar.jar agent -f index.conf

最后来个截图：

0 1