elk 日志分析系统
Logstash+ElasticSearch+Kibana4
- logstash 管理日志和事件的工具
- ElasticSearch 搜索
- Kibana4 功能强大的数据显示客户端
- redis 缓存
安装包
- logstash-1.4.2-1_2c0f5a1.noarch.rpm
- elasticsearch-1.4.4.noarch.rpm
- logstash-contrib-1.4.2-1_efd53ef.noarch.rpm
- kibana-4.0.1-linux-x64.tar.gz
安装jdk
openjdk或者Oracle的jdk都可以。
这里用openjdk
<code class="hljs cmake has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">yum <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">install</span> java-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.7</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-openjdk</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
安装redis
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">yum install redis/etc/init<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.d</span>/redis start查看redis中的键值redis-<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">cli</span> keys <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'*'</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>
安装测试ElasticSearch
<code class="hljs java has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">rpm -ivh elasticsearch-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.4</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.4</span>.noarch.rpm确认<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9200</span>端口监听有正常返回值即可:curl -X GET http:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//localhost:9200</span>[root<span class="hljs-annotation" style="color: rgb(155, 133, 157); box-sizing: border-box;">@iZ</span>28ywqw7nhZ ~]# curl -X GET http:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//localhost:9200</span>{ <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"status"</span> : <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">200</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"name"</span> : <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Damballah"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"cluster_name"</span> : <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"elasticsearch"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"version"</span> : { <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"number"</span> : <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"1.4.4"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"build_hash"</span> : <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"c88f77ffc81301dfa9dfd81ca2232f09588bd512"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"build_timestamp"</span> : <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"2015-02-19T13:05:36Z"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"build_snapshot"</span> : <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">false</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"lucene_version"</span> : <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"4.10.3"</span> }, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"tagline"</span> : <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"You Know, for Search"</span>}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li></ul>安装logstash
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">rpm -ivh logstash-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.4</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>_2c0f5a1<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.noarch</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.rpm</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
logstash配置
最简单的是接受一个输入，然后将在输出出来:
<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bin/logstash <span class="hljs-operator" style="box-sizing: border-box;">-e</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'input { stdin { } } output { stdout {} }'</span>helo<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">03</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">19</span>T09:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38.161</span>+<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0000</span> iZ28ywqw7nhZ helo</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>类似的还有:
<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bin/logstash <span class="hljs-operator" style="box-sizing: border-box;">-e</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'input { stdin { } } output { stdout { codec => rubydebug } }'</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>但是上面两个并没有很大的实际意义，我们可以将数据插入到elasticsearch中，然后用kibana显示出来。
- 首先要确保elasticsearch启动，9200监听。
- 然后插入数据:
<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">/opt/logstash/bin/logstash <span class="hljs-operator" style="box-sizing: border-box;">-e</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'input { stdin { } } output { elasticsearch { host => localhost } }'</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>然后可以用kibana 查看
或者访问
curl ‘http://localhost:9200/_search?pretty’ 来查看数据
下载kibana，运行
kibana就是一个java包。
<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">tar <span class="hljs-attribute" style="box-sizing: border-box;">-xf</span> kibana<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">-</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4.0</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-attribute" style="box-sizing: border-box;">-linux</span><span class="hljs-attribute" style="box-sizing: border-box;">-x64</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>tar<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>gzbin/kibana <span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">></span> kibana<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">log</span> <span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">&</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
里面配置index patterns ，用默认的就行了。
后记
基本上elk都是连着用的，连官网都是同一个官方网站，比如说elasticsearch一般是从logstash上面取数据的。kibana 默认又是从elasticsearch上面取数据。三者结合的非常好。
更多功能正在探索中。elk 很强大的数据收集查询分析开源解决方案。
http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash
http://www.cnblogs.com/buzzlight/p/logstash_elasticsearch_kibana_log.html
Logstash 最佳实践
Upgrade Required Your version of Elasticsearch is too old. Kibana requires Elasticsearch 0.90.9 or above.
init: tty (/dev/tty4) main process killed by KILL signal
list all indexs
curl ‘localhost:9200/_cat/indices?v’
delete
curl -XDELETE ‘localhost:9200/customer?pretty’
curl ‘localhost:9200/_cat/indices?v’
http://www.elastic.co/guide/en/elasticsearch/reference/1.x/_index_and_query_a_document.html
0 0
