使用logstash将kafka数据入到elasticsearch

logstash

安装

wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.1.tar.gz

 

tar zxf logstash-5.5.1.tar.gz

 

cd logstash-5.5.1

 

新建配置文件

vi config/kafka-logstash-es.conf

如下内容：

input {

    kafka {

        bootstrap_servers=> "192.168.101.200:9092"

        group_id =>"es"

        topics =>"topic123"

        consumer_threads =>1

        decorate_events =>true

        }

    }

output {

    elasticsearch {

        hosts =>"192.168.101.200:9200"

    index =>"topic123"

          }

     }

 

多个input如下格式：

 

input {

    kafka {

        bootstrap_servers=> "192.168.101.200:9092"

        group_id =>"es"

        topics =>"topic123"

        consumer_threads =>1

        decorate_events =>true

        type =>"topic123"

        }

    kafka {

        bootstrap_servers=> "192.168.101.200:9092"

        group_id =>"es"

        topics =>"topic1"

        consumer_threads =>1

        decorate_events =>true

        type =>"topic1"

        }

    }

output {

        if[type] =="topic123"{

                elasticsearch{

                        hosts=> "192.168.101.200:9200"

                        index=> "topic123"

                }

        }else{

                elasticsearch{

                hosts =>"192.168.101.200:9200"

                index =>"topic1"

          }

        }

     }

安装kafka插件(不确定是否必须)

bin/logstash-plugin install logstash-input-kafka

启动

bin/logstash -f config/kafka-logstash-es.conf–debug

 

参考资料：

logstash 官方手册 （参数配置等都看这个）

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html