Filebeat + Elasitcsearch + Kibana 日志收集系统

Filebeat 5.6.3

下载地址：https://www.elastic.co/downloads/beats/filebeat
安装在需要搜集的日志机器

1.配置filebeat，vim  /opt/software/filebeat-5.6.3-linux-x86_64/filebeat.yml

paths:

    #- /var/log/*.log

    - /data/ball/dirmap/application/log/resin/access.log     （本用例，搜集nginx日志）

fields:

    host: 172.16.52.89

    fields_under_root: true

output.elasticsearch:

  # Array of hosts to connect to.

  hosts: ["172.16.52.85:9200"]

tail_files: true

2.启动filebeat

./filebeat -e -c filebeat.yml

Elasitcsearch 5.6.3

下载地址：https://www.elastic.co/downloads/elasticsearch

安装在日志服务器

遇到的问题：

1.配置elasticsearch

1.1问题can not run elasticsearch as root

adduser elasticsearch创建elasticsearch用户

passwd elasticsearch修改elasticsearch密码

vim /etc/sudoers 赋予root权限

添加elasticsearch    ALL=(ALL)       ALL

        chown -R elasticsearch /opt/soft/elasticsearch  将安装目录设置为elasticsearch 

1.2.问题max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

vim /etc/sysctl.conf     

添加vm.max_map_count=655360

退出vim执行，sysctl -p，让配置生效

1.3.问题max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]

vim /etc/security/limits.conf 

添加 #*                soft     nofile          65536

#*                hard    nofile          65536

退出vim，重新ssh启动。

1.4.无法访问http://172.16.52.85:9200

开启IP 0.0.0.0使其通过其他机器访问

vim   /opt/soft/elasticssearch/conf/elasticsearch.yml

添加network.host: 0.0.0.0

2.启动filebeat

/opt/software/elasticsearch-5.6.3/bin/elasticsearch

浏览器访问http://172.16.52.85:9200/

Kibana5.6.3

下载地址：https://www.elastic.co/downloads/kibana

安装在日志服务器

1.启动/opt/soft/kibana-5.6.3-linux-x86_64/bin/kibana

 2.访问http://172.16.52.85:5601

3.配置index

4.查看nginx日志

推荐几个关于LEK的博客

官网

https://www.elastic.co/cn/products

博客

http://www.cnblogs.com/aarond/p/logstash.html

http://www.ruanyifeng.com/blog/2017/08/elasticsearch.html

http://www.cnblogs.com/wangxiaoqiangs/p/5798565.html

http://www.cnblogs.com/davidgu/p/6910851.html

http://www.dahouduan.com/2016/10/17/bigdata-filebeat-elasticsearch-kibana-elk/

http://www.cnblogs.com/zlslch/p/6622079.html