hive写入Elasticsearch参数设置

来源:互联网 发布:php.ini没有extension 编辑:程序博客网 时间:2024/05/22 01:34


1. 在hive中增加number of Hadoop input splits

Reduce the input split size from the default value. The mappers will get increased.

SET mapreduce.input.fileinputformat.split.maxsize;

参考:http://stackoverflow.com/questions/30222032/increase-number-of-hive-mappers-in-hadoop-2

参考:https://hadoopjournal.wordpress.com/2015/06/13/set-mappers-in-pig-hive-and-mapreduce/


2. Elasticsearch-Hadoop参数

es.batch.size.bytes (default 1mb)

es.batch.size.entries (default 1000)

es.batch.write.refresh (default true)

参考:https://www.elastic.co/guide/en/elasticsearch/hadoop/current/performance.html#_decrease_bulk_size

参考:https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html


0 0