Elasticsearch、Logstash、Kibana、Filebeat的使用总结

ELK是什么？
ELK Stack是软件集合Elasticsearch、Logstash、Kibana的简称，由这三个软件及其相关的组件可以打造大规模日志实时处理系统。

ElasticSearch：是一个基于Lucene的搜索服务器。它提供了一个分布式多用户能力的全文搜索引擎，基于RESTful web接口。Elasticsearch是用Java开发的，并作为Apache许可条款下的开放源码发布，是当前流行的企业级搜索引擎。设计用于云计算中，能够达到实时搜索，稳定，可靠，快速，安装使用方便。

Logstash：是一款强大的数据处理工具，它可以实现数据传输，格式处理，格式化输出，还有强大的插件功能，常用于日志处理。

Kibana：是一个可视化工具，主要负责查询 Elasticsearch 的数据并以可视化的方式展现给业务方，比如各类饼图、直方图、区域图等。

关于Elasticsearch的版本，我是从5.x版本开始学起的，听说之前的版本跳跃2.x--->5.x,貌似变化还挺大的。关于head插件在5.x之后变成独立服务了。

关于ES集群的安装配置以及head插件的安装参考：http://blog.csdn.net/chenxun_2010/article/details/78437852

logstash最佳实践：https://doc.yonyoucloud.com/doc/logstash-best-practice-cn/get_start/hello_world.html

Logstash使用：
https://www.elastic.co/guide/en/logstash/current/pipeline.html

Logstash 工作的三个阶段： input、filter、output

安装logstash： 其安装非常简单，只需要下载安装包解压开箱即用。 只要写配置文件即可。
启动方式：

./bin/logstash   -f    your_config.file

配置文件的写法格式：参考http://blog.csdn.net/chenxun_2010/article/details/78605934

input {    file {        path => ["/home/elk_test/logstash-5.6.3/logfile"]        codec => json {            charset => "UTF-8"        }    }}output {    stdout {        codec => rubydebug    }    elasticsearch {        hosts => "192.168.0.153:9200"    }}

Kibana的安装也非常简单，下载安装包解压修改一下配置文件就可以：

server.host: "192.168.2.181"#elasticsearch.username: "elastic"#elasticsearch.password: "changeme"elasticsearch.url: "http://192.168.2.181:9200"kibana.index: ".kibana"

Filebeat：的使用
filebeat：部署在具体的业务机器上，通过定时监控的方式获取增量的日志，并转发到logstash、elasticsearch、kafka等等。

配置vim filebeat.yml

#=========================== Filebeat prospectors =============================filebeat.prospectors:# Each - is a prospector. Most options can be set at the prospector level, so# you can use different prospectors for various configurations.# Below are the prospector specific configurations.- input_type: log  # Paths that should be crawled and fetched. Glob based paths.  paths:    - /home/elk/logfile    #- c:\programdata\elasticsearch\logs\*

输出到logstash：

#----------------------------- Logstash output --------------------------------output.logstash:  # The Logstash hosts  hosts: ["192.168.2.181:5044"]

logstash的配置

input {    beats {        host => "192.168.2.181"        port => 5044     codec => json}}output {    stdout {        codec => rubydebug    }    elasticsearch {        hosts => "192.168.2.181:9200"    }}

启动filebeat：

./filebeat

启动logstash：

./bin/logstash   -f   beat.config

把准备好的json数据输入到filebeat监控的文件中去：

{ "firstName": "1", "lastName":"McLaughlin", "email": "aaaa" }{ "firstName": "2", "lastName":"Hunter", "email": "bbbb"}{ "firstName": "3", "lastName":"Harold", "email": "cccc" }

在logstash端我们看到输出：成功解析json字段

{     "firstName" => "1",      "lastName" => "McLaughlin",    "@timestamp" => 2017-12-01T08:38:37.480Z,        "offset" => 63,      "@version" => "1",          "beat" => {            "name" => "Ubuntu-20170424",        "hostname" => "Ubuntu-20170424",         "version" => "5.6.3"    },    "input_type" => "log",          "host" => "Ubuntu-20170424",        "source" => "/home/elk/filebeat-5.6.3-linux-x86_64/request",          "type" => "log",         "email" => "aaaa",          "tags" => [        [0] "beats_input_codec_json_applied"    ]}{     "firstName" => "2",      "lastName" => "Hunter",    "@timestamp" => 2017-12-01T08:39:02.482Z,        "offset" => 121,      "@version" => "1",          "beat" => {            "name" => "Ubuntu-20170424",        "hostname" => "Ubuntu-20170424",         "version" => "5.6.3"    },    "input_type" => "log",          "host" => "Ubuntu-20170424",        "source" => "/home/elk/filebeat-5.6.3-linux-x86_64/request",          "type" => "log",         "email" => "bbbb",          "tags" => [        [0] "beats_input_codec_json_applied"    ]}{     "firstName" => "3",      "lastName" => "Harold",    "@timestamp" => 2017-12-01T08:39:02.482Z,        "offset" => 180,      "@version" => "1",          "beat" => {            "name" => "Ubuntu-20170424",        "hostname" => "Ubuntu-20170424",         "version" => "5.6.3"    },    "input_type" => "log",          "host" => "Ubuntu-20170424",        "source" => "/home/elk/filebeat-5.6.3-linux-x86_64/request",          "type" => "log",         "email" => "cccc",          "tags" => [        [0] "beats_input_codec_json_applied"    ]}

在head插件管理页面看到数据：