Elasticsearch & plugins(ELK+kafka+filebeat) 安装教程
来源:互联网 发布:java从入门到放弃系列 编辑:程序博客网 时间:2024/03/29 19:45
一、Elasticsearch 安装文档
环境准备
CentOS7.1JDK1.8elasticsearch-5.4.3
###1.1 JDK安装配置wget -c http://download.oracle.com/otn-pub/java/jdk/8u111-b14/jdk-8u111-linux-x64.tar.gz
###1.2 elasticsearch安装####1.2.1 下载解压cd 指定目录 #指定下载目录wget -N https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.4.3.tar.gztar -zxf elasticsearch-5.4.3.tar.gzln -s elasticsearch-5.4.3 elasticsearchmkdir -p /var/data/elasticsearchmkdir -p /var/logs/elasticsearch
####1.2.2 修改配置文件vi elasticsearch/config/elasticsearch.ymlcluster.name: elasticstackpath.data: /var/data/elasticsearchpath.logs: /var/logs/elasticsearchnetwork.host: 0.0.0.0http.port: 11200transport.tcp.port: 11300discovery.zen.ping.unicast.hosts: ["10.213.162.77", "10.213.162.78", "10.213.162.79"]discovery.zen.minimum_master_nodes: 3http.cors.enabled: truehttp.cors.allow-origin: "*"
####1.2.3 创建elasticsearch用户elasticsearch不能用root用户启动groupadd elasticsearchuseradd elasticsearch -g elasticsearch -p elasticsearch
####1.2.4 修改服务器相关参数修改vm.map 限制sysctl -w vm.max_map_count=262144或vi /etc/sysctl.confvm.max_map_count=262144修改文件限制ulimit -n 102400或vi /etc/security/limits.confelasticsearch hard nofile 102400elasticsearch soft nofile 102400
####1.2.5 切换到elasticsearch用户下启动su elasticsearchcd /var/wd/elasticsearchchown -R elasticsearch /var/wd/elasticsearchchown -R elasticsearch /var/datachown -R elasticsearch /var/logs./bin/elasticsearch
验证是否启动是否成功curl ‘http://10.213.162.77:11200’##二、head插件安装文档###2.1 安装nodecd /usr/localwget -N https://nodejs.org/dist/v7.2.0/node-v7.2.0-linux-x64.tar.gztar -zxf node-v7.2.0-linux-x64.tar.gzln -s node-v7.2.0-linux-x64 nodevi /etc/profileexport PATH=$PATH:/usr/local/node/binsource /etc/profile
###2.2 安装gruntnpm install -g grunt-cli
###2.3 安装head####2.3.1 下载head插件源码git clone git://github.com/mobz/elasticsearch-head.git
####2.3.2 修改Gruntfile.jsconnect: { server: { options: { port: 11100, hostname: '*', base: '.', keepalive: true } }}
####2.3.3 修改app.jsthis.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://10.213.162.77:12200";
####2.2.4 运行headnpm install grunt server
##三、Kibana安装文档###3.1 下载及安装注:Kibana必须保证和elasticsearch版本一致###3.1.1 下载cd /var/wdwget -N https://artifacts.elastic.co/downloads/kibana/kibana-5.4.3-linux-x86_64.tar.gztar -zxf kibana-5.4.3-linux-x86_64.tar.gzln -s kibana-5.4.3 kibana
###3.1.2 修改配置vi kibana/config/kibana.ymlserver.port: 11201server.host: "0.0.0.0"elasticsearch.url: "http://10.213.162.78:11200"elasticsearch.username: "elastic"elasticsearch.password: "elastic"
##四、Logstash安装 Logstash 是一款强大的数据处理工具,它可以实现数据传输,格式处理,格式化输出,还有强大的插件功能,常用于日志处理。###4.1 logstash版本要求| Kafka Client Version | Logstash Version | Plugin Version | Why? ||———-|———–|———-|——————————-|| 0.8 |2.0.0 - 2.x.x | ###4.2 logstash下载安装cd /var/wd/wget -c https://artifacts.elastic.co/downloads/logstash/logstash-5.4.3.tar.gztar -xzvf logstash-5.4.3.tar.gzln -s logstash-5.4.3 logstashcd logstash
###4.3 日志采集filebeat配置mkdir plugin-configvi plugin-config/filebeat.conf
###4.4 日志采集filebeat配置input { beats { port => "10044" }}# The filter part of this file is commented out to indicate that it is# optional.filter { #grok根据日志格式配置 grok { match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{WORD:trace_id} \[.*\] %{LOGLEVEL:level}"] remove_field => [ "beat","tags"] } }output { #logstash直接输出到es elasticsearch { hosts => ["10.213.131.131:11200","10.213.131.132:11200","10.213.131.134:11200"] index => "%{[@metadata][beat]}-%{+YYYY.MM}" document_type => "%{[@metadata][type]}" }}
这里也可以通过kafka来缓存日志,减少logstash直连kafka压力这里logstash接受filebeat采集日志发送到kafkainput { beats { port => "10088" }}# The filter part of this file is commented out to indicate that it is# optional.filter { grok { match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{WORD:trace_id} \[.*\] %{LOGLEVEL:level}"] remove_field => [ "beat","tags"] } }#output {# stdout { codec => rubydebug }# elasticsearch {# hosts => ["10.213.131.131:11200","10.213.131.132:11200","10.213.131.134:11200"]# index => "%{[@metadata][beat]}-%{+YYYY.MM}"# document_type => "%{[@metadata][type]}"# }#}output { kafka {# codec => plain {# format => "%{message}"# } codec => json topic_id => "topic" bootstrap_servers => "127.0.0.1:12015" #kafka brokers compression_type => "snappy" #c++压缩,快速 }}
这里logstash接受kafka日志写elasticsearchinput{ kafka{ codec => "json" group_id => "feeds-log-logstash" topics => ["feeds-log"] bootstrap_servers => "127.0.0.1:12015" #kafka brokers }}output{ elasticsearch{ hosts=>["10.213.131.131:11200","10.213.131.132:11200","10.213.131.132:11200"] index => "flog-%{+YYYYMM}" document_type => "%{[fields][app]}" #flush_size=>20000 #idle_flush_time=>10 #template_overwrite=>true }}
###4.5 启动logstash 控台启动,观察错误日志,没问题在后台启动bin/logstash -f plugin-config/filebeat.conf --config.reload.automatic
##五、Filebeat安装###5.1 Filebeat介绍Beats 平台是 Elastic.co 从 packetbeat 发展出来的数据收集器系统。beat 收集器可以直接写入 Elasticsearch,也可以传输给 Logstash。其中抽象出来的 libbeat,提供了统一的数据发送方法,输入配置解析,日志记录框架等功能。也就是说,所有的 beat 工具,在配置上,除了 input 以外,在output、filter、shipper、logging、run-options 上的配置规则都是完全一致的 ,filebeat是beat中的一员。
5.2 Filebeat下载
cd /var/wd/wget -c https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.4.3-linux-x86_64.tar.gztar -xzvf filebeat-5.4.3-linux-x86_64.tar.gzcd filebeat-5.4.3-linux-x86_64
这里安装的是bit版,也可以选择rpm版本安装
wget -c https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.4.3-x86_64.rpm
5.3 Filebeat配置
vi filebeat.yml
5.3.1 采集数据源
- input_type: log # Paths that should be crawled and fetched. Glob based paths. paths: - /var/*.log
5.4.2 添加产生日志应用名
fields: app: XXXX
5.4.3 指定输出
output.logstash: # The Logstash hosts hosts: ["10.213.131.132:10044","10.213.131.131:10044"] worker: 2 loadbalance: true index: feeds-log
5.4.3 启动关闭脚本
- 启动脚本
vi startup.sh#!/bin/bashnohup ./filebeat -e -c filebeat.yml -d publish &
- 关闭脚本
vi shutdown.sh#!/bin/bashrunningPID=`pgrep -f "./filebeat -e -c filebeat.yml -d publish"`if [ "$runningPID" ]; then echo "filebeat pid: $runningPID" kill -15 $runningPIDfisleep 2
阅读全文
0 0
- Elasticsearch & plugins(ELK+kafka+filebeat) 安装教程
- ELK + Kafka + Filebeat学习
- ELK(Elasticsearch,logstash,kibana)无痛安装教程
- ELK中filebeat组件的安装-04
- Docker ELK+Filebeat安装与配置
- ELK+Filebeat的安装和联调
- CENTOS7 ELK 5.3.2 elasticsearch logstash kibana filebeat
- ELK+Filebeat+Kafka+ZooKeeper 构建海量日志分析平台
- ELK+Filebeat+Kafka+ZooKeeper 构建海量日志分析平台
- elk日志分析filebeat配置(filebeat + logstash)
- elk的安装部署三(kibana的安装及使用filebeat收集日志)
- linux ELK(filebeat)环境搭建中加入kafka消息队列的详细介绍(菜鸟新手级别)
- ELK log4j kafka logstash elasticsearch系统搭建
- filebeat+kafka+ELK5.4安装与部署
- ELK(二)ElasticSearch
- ELK之Filebeat安装与配置及使用
- Elasticsearch ——ELK安装
- ELK第一篇:Elasticsearch安装
- uoj274 温暖会指引我们前行
- “无法找到xxx.exe的调试信息,或者调试信息不匹配。未调用调试信息生成二进制文件……”问题的解决办法
- Java运算符
- Java中的三种内部类
- 关于major和无人驾驶的一点想法(写给一个微信群)
- Elasticsearch & plugins(ELK+kafka+filebeat) 安装教程
- textfield datepicker作为响应者、自定义toolbar方法
- OAI搭建之SIM卡
- 欢迎登陆页面!
- 解决:ubuntu16.04启动时长时间停留在紫屏或跳文本的黑屏界面
- CCF201409-3 字符串匹配 初级KMP 一句判断大小写锁定
- 中译日练习
- STM8S003F3通过PWM波实现三基色呼吸灯
- 文章标题