从零搭建ELK实时日志分析平台(ElasticSearch, Logstash, Kibana)

前言：先说说搭建这个平台的环境吧
系统：centos7
jdk：1.8
ElasticSearch：5.5.2
Logstash：5.3.2
Kibana：5.2.2

何谓从零，就是新建一个centos7开始
1、获取ip addr 这里我假设是192.168.12.128
2、关闭防火墙：systemctl stop firewalld.service
3、安装与配置jdk1.8
可以参考以下：http://blog.csdn.net/sinat_15153911/article/details/77478850
配置环境记住要用$符号：

JAVA_HOME=/home/jdk/jdk1.8.0_131PATH=$JAVA_HOME/bin:$PATHCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexport NODE_HOME=/home/nodejs/node-v8.9.3-linux-x64export PATH=$NODE_HOME/bin:$PATHexport PATH USER...

4、安装与配置ElasticSearch
之前是安装在windows玩的，http://blog.csdn.net/sinat_15153911/article/details/78118200 由于百度云的问题是下载不了代码的，需要的请加QQ490647751开通vip获取。

network.host: 192.168.12.128http.port: 9200

访问：http://192.168.12.128:9200

正常如下图例子所示：下图是6.1.0版本的图片，我们配置的版本是5.5.2

问题：
不能用root打开：

[root@bogon ~]# useradd elastic[root@bogon ~]# chown -R elastic:elastic /home/es/elasticsearch-5.5.2/新建elastic用户 并且把目录权限赋予给elastic 

ERROR: [2] bootstrap checks failed

[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

问题1 [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]修改/etc/security/limits.conf文件，添加或修改如下行： （请切换到root用户 然后强制修改文件）*        hard    nofile           65536*        soft    nofile           65536问题2  [2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]$ sudo sysctl -w vm.max_map_count=2621441或者修改 /etc/sysctl.conf 文件，添加 “vm.max_map_count”设置 永久改变(sudo sysctl -p /etc/sysctl.conf生效)。

4.1、安装和配置head
4.1.1 安装nodejs

tar -xJf node-v8.9.1-linux-x64.tar.xz然后我们再配置环境变量：vi /etc/profile

[root@localhost node-v8.9.3-linux-x64]# node -vv8.9.3[root@localhost node-v8.9.3-linux-x64]# npm -v5.5.1

4.1.2 安装git
yum install -y git
git –version

4.1.3 安装head

git clone git://github.com/mobz/elasticsearch-head.gitcd elasticsearch-headnpm installnpm run startopen http://localhost:9100/

进入elasticsearch config目录 打开 elasticsearch.yml

最后加上

http.cors.enabled: truehttp.cors.allow-origin: "*"

测试

启动elasticsearch，再进入head目录，执行npm run start 启动插件

说明启动成功，然后浏览器 执行 http://192.168.12.128:9100/

5、安装与配置logstash

http.host: “192.168.12.128”
http.port: 9600

我把它的log4j 删了，那里的日志文件不存在什么的
然后加一个log4j_to_es.conf文件

input {  log4j {    mode => "server"    host => "192.168.12.128"    port => 4560  }}filter {  #Only matched data are send to output.}output {    elasticsearch {    action => "index"          #The operation on ES    hosts  => "192.168.12.128:9200"   #ElasticSearch host, can be array.    index  => ".kibana"         #The index to write data to.  }}

打开方式：./logstash -f config/log4j_to_es.conf

6、安装与配置Kibana

配置：

server.port: 5601server.host: "192.168.12.128"elasticsearch.url: "http://192.168.12.128:9200"kibana.index: ".kibana"

要输入之前配置的这个 .kibana

7、springboot log test

 <dependency>            <groupId>org.springframework.boot</groupId>            <artifactId>spring-boot-starter</artifactId>            <version>1.3.8.RELEASE</version>            <exclusions>                <exclusion>                    <groupId>org.springframework.boot</groupId>                    <artifactId>spring-boot-starter-logging</artifactId>                </exclusion>            </exclusions>        </dependency>        <dependency>            <groupId>org.springframework.boot</groupId>            <artifactId>spring-boot-starter-log4j</artifactId>            <version>1.3.8.RELEASE</version>        </dependency>        <dependency>            <groupId>org.springframework.boot</groupId>            <artifactId>spring-boot-starter-test</artifactId>            <version>1.3.8.RELEASE</version>            <scope>test</scope>        </dependency>

log4j.rootLogger=INFO,console# for package com.demo.elk, log would be sent to socket appender.log4j.logger.com.forezp=DEBUG, socket# appender socketlog4j.appender.socket=org.apache.log4j.net.SocketAppenderlog4j.appender.socket.Port=4560log4j.appender.socket.RemoteHost=192.168.12.128log4j.appender.socket.layout=org.apache.log4j.PatternLayoutlog4j.appender.socket.layout.ConversionPattern=%d [%-5p] [%l] %m%nlog4j.appender.socket.ReconnectionDelay=10000# appender consolelog4j.appender.console=org.apache.log4j.ConsoleAppenderlog4j.appender.console.target=System.outlog4j.appender.console.layout=org.apache.log4j.PatternLayoutlog4j.appender.console.layout.ConversionPattern=%d [%-5p] [%l] %m%n

需要源码学习的，可以加QQ490647751回复‘开通vip获取从零搭建ELK实时日志分析平台(ElasticSearch, Logstash, Kibana)’。