How to Set Up the ELK Stack- Elasticsearch, Logstash and Kibana
来源:互联网 发布:杰克琼斯淘宝旗舰店 编辑:程序博客网 时间:2024/05/03 16:55
http://knowm.org/how-to-set-up-the-elk-stack-elasticsearch-logstash-and-kibana/
Elastic Search, Logstash and Kibana – the ELK Stack – is emerging as the best technology stack to collect, manage and visualize big data. If you came here looking for help installing the ELK stack, you don’t need an introduction, so let’s get right down to the dirty work. The following guide shows how to install Java 8, Elasticsearch 2.3, Logstash 2.3 and Kibana 4 on Ubuntu with init.d (system v
) or alternatively with systemd
. You can do one or the other depending on your system and/or preferences. In two previous posts Integrate Bro IDS with ELK Stack and How to Install Bro Network Security Monitor on Ubuntu
, we showed how to install Bro and parse the generated Bro logs with Logstash. With the entire stack installed, running, and parsing logs generated by Bro, Kibana allows for a wonderful data analysis and discovery process. Of course, almost any data source can be used, and not just Bro.
Oracle Java 8
Elasticsearch
Note: Check for the latest Elastic Search release version here: downloads/elasticsearch
System V
Systemd
Configure
Note: If you want to access your Elasticsearch instance from clients on a different IP address via Javascript, add the following inside elasticsearch.yml
:
Also note that if you want to access Elasticsearch
of any of the plugins like kopf
from a host besides local host, you’ll need to add the following to elasticsearch.yml
:
FYI, the Elasticsearch stores your actual data in /var/lib/elasticsearch/elasticsearch/nodes/...
.
Test
In browser: http://localhost:9200/
Hello World Data
Debugging
Debug startup errors by running elasticsearch in the console
Elasticsearch Kopf Plugin (an aside)
The kopf plugin provides an admin GUI for Elasticsearch. It helps in debugging and managing clusters and shards. It’s really easy to install (check here for latest verion):
View in browser at: http://localhost:9200/_plugin/kopf/#!/cluster. You should see something like this:
Logstash
Note: Check for the latest Logstash release version here: downloads/logstash
System V
Systemd
Configure
By default Logstash filters will only work on a single thread, and thus also one CPU core. To increase the number of cores available to LogStash, edit the file /etc/default/logstash and set the -w parameter to the number of cores:LS_OPTS="-w 8"
.
You can increase the Java heap size here as well. Make sure to uncomment the line you are updating. Don’t forget to restart logstash afterwards.
Test
Directly change java options in script if you are starting logstash
from the command line and not as a linux service.
Hello World (warning Logstash at the command line is slow to start, so be patient)
Hello World with Elastic Search
Plugins
For non-standard parsing features, we access plugins. The following terminal commands show how to install thelogstash-filter-translate plugin. For a more in-depth explanation of installing logstash
plugins see How to Install Logstash Plugins for Version 1.5.
Kibana
Note: Check for the latest Kibana release version here: downloads/kibana
System V
Systemd
Configure
Test
In browser: http://localhost:5601
Final Words
If all went well, the next step is to tap into a datasource with Logstash and view it with Kibana. In two previous postsIntegrate Bro IDS with ELK Stack and How to Install Bro Network Security Monitor on Ubuntu
, we showed how to install Bro and parse the generated Bro logs with Logstash. The following is a screen shot from a Kibana dashboard we made for one of our websites bitcoinium.com showing some nice bar and pie charts. Once everything is set up and running it immediately becomes clear how useful the ELK stack is.
Related Resources
Buy the Plug and Play Network Monitor directly from knowm.org: http://knowm.org/product/plug-and-play-network-monitor/
Integrating Bro with the ELK Stack: http://knowm.org/integrate-bro-ids-with-elk-stack/
How to Created a Bonded Network Interface: http://knowm.org/how-to-create-a-bonded-network-interface/
- How to Set Up the ELK Stack- Elasticsearch, Logstash and Kibana
- How To Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Ubuntu 14.04
- How To Install Elasticsearch 1.7, Logstash 1.5, and Kibana 4.1 (ELK Stack) on Ubuntu 14.04
- ELK-ElasticSearch+Logstash+Kibana
- 在 Ubuntu14.04上安装 Elasticsearch, Logstash, Kibana (ELK Stack)
- Elasticsearch, Logstash, Kibana (ELK) 搭建
- ELK=Elasticsearch+Logstash+kibana
- ELK(Elasticsearch,logstash,kibana)无痛安装教程
- ELK(elasticsearch+logstash+kibana) 日志分析
- kubernetes 1.5安装 ELK(ElasticSearch, Logstash, Kibana)
- ELK(ElasticSearch, Logstash, Kibana)实现分布式日志
- ELK (Elasticsearch+Logstash+Kibana) 的安装
- ELK(ElasticSearch, Logstash, Kibana)搭建日志分析平台-Kibana篇
- ELK(ElasticSearch, Logstash, Kibana)搭建日志分析平台-Logstash篇
- ELK(ElasticSearch, Logstash, Kibana)搭建日志分析平台-Elasticsearch篇
- 最近ELK(elasticsearch+logstash+kibana)学习小结
- ELK(ElasticSearch, Logstash, Kibana)搭建实时日志分析平台
- ELK(ElasticSearch, Logstash, Kibana)搭建实时日志分析平台
- 配置
- WSDL文件生成
- 快速傅里叶变换 FFT 【bzoj2194】 快速傅立叶之二
- 244. Shortest Word Distance II
- 基于JAX-WS的webService开发实例
- How to Set Up the ELK Stack- Elasticsearch, Logstash and Kibana
- 母函数(Generating function)详解 — TankyWoo
- PHP如何实现网址伪静态
- tlb原理1
- 电子商城开发摘记
- [易飞]自制成本调整单金额与数量测算小工具
- 让nginx支持.htaccess文件实现伪静态的方法!
- PHP垃圾回收深入理解
- 【MongoDb探究】06-深入查询表达式2