elk 搭建
来源：互联网 发布：游戏性的gal 知乎 编辑：程序博客网 时间：2024/05/09 11:23
elk 平台搭建:ELK平台搭建系统环境System: Centos release 6.7 (Final)ElasticSearch: 2.1.0Logstash: 2.1.1Kibana: 4.3.0Java: openjdk version "1.8.0_65"注：由于Logstash的运行依赖于Java环境， 而Logstash 1.5以上版本不低于java 1.7，因此推荐使用最新版本的Java。因为我们只需要Java的运行环境，所以可以只安装JRE，不过这里我依然使用JDK，请自行搜索安装。配置ElasticSearch：tar -zxvf elasticsearch-2.1.0.tar.gzcd elasticsearch-2.1.0zjtest7-redis:/usr/local/elasticsearch-2.3.4# ./bin/plugin install mobz/elasticsearch-head-> Installing mobz/elasticsearch-head...Plugins directory [/usr/local/elasticsearch-2.3.4/plugins] does not exist. Creating...Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ...Downloading .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONEVerifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ...NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)Installed head into /usr/local/elasticsearch-2.3.4/plugins/head然后编辑ES的配置文件：vi config/elasticsearch.yml添加配置:cluster.name=es_clusternode.name=node0path.data=/tmp/elasticsearch/datapath.logs=/tmp/elasticsearch/logs#当前hostname或IP，我这里是centos2network.host=192.168.32.80network.port=9200启动报错:zjtest7-redis:/usr/local/elasticsearch-2.3.4# ./bin/elasticsearchException in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: ElasticsearchParseException[malformed, expected settings to start with 'object', instead was [VALUE_STRING]];Likely root cause: ElasticsearchParseException[malformed, expected settings to start with 'object', instead was [VALUE_STRING]]at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:65)at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)at org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1080)at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1067)at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:107)at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:100)at org.elasticsearch.bootstrap.BootstrapCLIParser.<init>(BootstrapCLIParser.java:48)at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:226)at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)Refer to the log for complete error details.原因 需要这么写:cluster.name: es_clusternode.name: node01path.data: /tmp/elasticsearch/datapath.logs: /tmp/elasticsearch/logsnetwork.host: 192.168.32.80network.port: 9200zjtest7-redis:/usr/local/elasticsearch-2.3.4# ./bin/elasticsearchException in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:93)at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:144)at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)Refer to the log for complete error details.创建 elk 用户和group:[elk@zjtest7-redis local]$ cd elasticsearch-2.3.4[elk@zjtest7-redis elasticsearch-2.3.4]$ ./bin/elasticsearch[2016-07-22 14:03:22,797][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in[2016-07-22 14:03:23,728][INFO ][node ] [node01] version[2.3.4], pid[1756], build[e455fd0/2016-06-30T11:24:31Z][2016-07-22 14:03:23,728][INFO ][node ] [node01] initializing ...[2016-07-22 14:03:25,787][INFO ][plugins ] [node01] modules [reindex, lang-expression, lang-groovy], plugins [head], sites [head][2016-07-22 14:03:25,878][INFO ][env ] [node01] using [1] data paths, mounts [[/ (/dev/mapper/vg00-lv_root)]], net usable_space [86.7gb], net total_space [96.2gb], spins? [possibly], types [ext4][2016-07-22 14:03:25,882][INFO ][env ] [node01] heap size [1015.6mb], compressed ordinary object pointers [true][2016-07-22 14:03:25,882][WARN ][env ] [node01] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536][2016-07-22 14:03:33,299][INFO ][node ] [node01] initialized[2016-07-22 14:03:33,302][INFO ][node ] [node01] starting ...[2016-07-22 14:03:33,561][INFO ][transport ] [node01] publish_address {192.168.32.80:9300}, bound_addresses {192.168.32.80:9300}[2016-07-22 14:03:33,576][INFO ][discovery ] [node01] es_cluster/gYNVInstR16CujdeJ1T6YQ[2016-07-22 14:03:36,717][INFO ][cluster.service ] [node01] new_master {node01}{gYNVInstR16CujdeJ1T6YQ}{192.168.32.80}{192.168.32.80:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)[2016-07-22 14:03:36,759][INFO ][http ] [node01] publish_address {192.168.32.80:9200}, bound_addresses {192.168.32.80:9200}[2016-07-22 14:03:36,759][INFO ][node ] [node01] started[2016-07-22 14:03:36,867][INFO ][gateway ] [node01] recovered [0] indices into cluster_state可以看到，它跟其他的节点的传输端口为9300，接受HTTP请求的端口为9200。使用ctrl+C停止。当然，也可以使用后台进程的方式启动ES：./bin/elasticsearch &然后可以打开页面localhost:9200，将会看到以下内容：http://192.168.32.80:9200/返回；{ "name" : "node01", "cluster_name" : "es_cluster", "version" : { "number" : "2.3.4", "build_hash" : "e455fd0c13dceca8dbbdbb1665d068ae55dabe3f", "build_timestamp" : "2016-06-30T11:24:31Z", "build_snapshot" : false, "lucene_version" : "5.5.0" }, "tagline" : "You Know, for Search"}返回展示了配置的cluster_name和name，以及安装的ES的版本等信息。刚刚安装的head插件，它是一个用浏览器跟ES集群交互的插件，可以查看集群状态、集群的doc内容、执行搜索和普通的Rest请求等。现在也可以使用它打开localhost:9200/_plugin/head页面来查看ES集群状态：LogstashLogstash的功能如下：[elk@zjtest7-redis logstash-2.3.4]$ ./bin/logstashNo command givenUsage: logstash <command> [command args]Run a command with the --help flag to see the arguments.For example: logstash agent --helpAvailable commands: agent - runs the logstash agent version - emits version info about this logstash[elk@zjtest7-redis logstash-2.3.4]$ ./bin/logstash agent -f config/log4j_to_es.confSettings: Default pipeline workers: 1log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAuthCache).log4j:WARN Please initialize the log4j system properly.log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.Pipeline main started[elk@zjtest7-redis logstash-2.3.4]$ ./bin/logstash agent -f config/log4j_to_es.conf &输入和输出日志:代码如下：input { file { type => "nginx_access" path => ["/usr/share/nginx/logs/test.access.log"] }}output { redis { host => "localhost" data_type => "list" key => "logstash:redis" }}output { # For detail config for elasticsearch as output, # See: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html elasticsearch { action => "index" #The operation on ES hosts => "192.168.32.80:9200" #ElasticSearch host, can be array. index => "applog" #The index to write data to. }}logstash,elasticsearch,kibana 怎么进行nginx的日志分析呢？首先，架构方面，nginx是有日志文件的，它的每个请求的状态等都有日志文件进行记录。其次，需要有个队列，redis的list结构正好可以作为队列使用。然后分析使用elasticsearch就可以进行分析和查询了。我们需要的是一个分布式的，日志收集和分析系统。logstash有agent和indexer两个角色。对于agent角色，放在单独的web机器上面，然后这个agent不断地读取nginx的日志文件，每当它读到新的日志信息以后，就将日志传送到网络上的一台redis队列上。对于队列上的这些未处理的日志，有不同的几台logstash indexer进行接收和分析。分析之后存储到elasticsearch进行搜索分析。再由统一的kibana进行日志web界面的展示。启动/usr/local/elasticsearch-2.3.4# ./bin/elasticsearch[elk@zjtest7-redis logstash-2.3.4]$ ./bin/logstash agent -f config/log4j_to_es.conflogstash日志收集分发到elastic集群，elasticsearch进行数据索引，kibana进行结构化查询展示，redis做缓存队列。装：logstash，新版下载即可用，需加配置文件agent.conf和index.conf，当然最好途径是官网sample即得即用。/usr/local/redis/bin/redis-server *:6379 input { file { type => "nginx_access" path => ["/usr/local/nginx/logs/test.access.log"] }}output { redis { host => "localhost" data_type => "list" key => "logstash:redis" port=>6379 password => "1234567" }}~ ~ zjtest7-redis:/usr/local/logstash-2.3.4# ./bin/logstash -f /usr/local/logstash-2.3.4/config/logstash_agent.conf Settings: Default pipeline workers: 1Pipeline main started日志格式:log_format logstash '$http_host $server_addr $remote_addr [$time_local] "$request" ' '$request_body $status $body_bytes_sent "$http_referer" "$http_user_agent" ' '$request_time $upstream_response_time';redis :127.0.0.1:6379> llen "logstash:redis"(integer) 328到这里数据已经存放到redis队列:logstash 数据写入到redis 队列:zjtest7-redis:/usr/local/logstash-2.3.4# ./bin/logstash -f /usr/local/logstash-2.3.4/config/logstash_agent.conf ### 存数据zjtest7-redis:/usr/local/logstash-2.3.4# ./bin/logstash -f /usr/local/logstash-2.3.4/config/logstash_indexer.conflogstash分为 index和aget ，agent负责监控、过滤日志，index负责收集日志并将日志交给ElasticSearch 做搜索Shipper：发送事件（events）至LogStash；通常，远程代理端（agent）只需要运行这个组件即可；Broker and Indexer：接收并索引化事件；Search and Storage：允许对事件进行搜索和存储；Web Interface：基于Web的展示界面192.168.32.162为日志查看服务器，该机器需要安装redis, elasticserch, logstatsh, kibana这四个应用程序。192.168.32.161?为应用程序nginx应用程序，我们这次只收集他的日志进行分析。这个配置文件,是读取nginx日志写入到rediszjtest7-redis:/usr/local/logstash-2.3.4/config# cat logstash_agent.conf input { file { type => "nginx_access" path => ["/usr/local/nginx/logs/test.access.log"] }}output { redis { host => "localhost" data_type => "list" key => "logstash:redis" port=>"6379" password => "1234567" }}这个配置文件是读取本地的redis数据,交给elasticsearchzjtest7-redis:/usr/local/logstash-2.3.4/config# cat logstash_indexer.conf input { redis { host => "localhost" data_type => "list" key => "logstash:redis" type => "redis-input" password => "1234567" port =>"6379" }}output { elasticsearch { embedded => false protocol => "http" host => "localhost" port => "9200" index => "access-%{+YYYY.MM.dd}" document_type="access" }stdout {codec => rubydebug}}Kibana配置Kibana:tar -zxvf kibana-4.3.0-linux-x86.tar.gzcd kibana-4.3.0-linux-x86vi config/kibana.ymlserver.port: 5601server.host: "192.168.32.80"elasticsearch.url: http://192.168.32.80:9200kibana.index:".kibana' 0 0
- elk 搭建
- ELK 搭建
- ELK搭建
- 搭建elk
- ELK搭建
- 搭建elk
- ELK+redis集群搭建
- 本地搭建ELK系统
- ELK搭建记录
- ELK平台的搭建
- ELK的搭建实践
- ELK环境的搭建
- 搭建ELK全过程
- ELK学习1：搭建
- 搭建ELK日志系统
- 第二节 elk搭建
- Mac上搭建ELK
- ELK环境搭建
- spring-jedis哨兵模式，不能远程连接
- 【杭电】[2141]Can you find it?
- Oracle中 instr 函数
- smali语法
- 如何判断jquery选择器选择结果为空
- elk 搭建
- Nexus6P使用小技巧
- 机器学习笔记第2周第6课octave教程和向量化
- 获取webservice 返回的 dataset 数据集 并转换成 datatable
- 安卓学习笔记---Intent的使用，如何改变Activity在当前任务堆栈中的顺序
- 孤儿进程与僵尸进程[总结]
- HDU 1429 胜利大逃亡(续)(BFS+状态压缩）
- dataTable 根据不同行数据动态显示行操作
- Android度量单位说明(DIP,DP,PX,SP)
