elasticsearch-jdbc插件使用

来源：互联网发布：怎样下载监控软件编辑：程序博客网时间：2024/05/16 04:16

最近用到elasticsearch-jdbc插件对数据进行导入，期间遇到一些小问题，进行记录下。

bin目录下的sh脚本配置如下：

bin=/e01/elasticsearch-jdbc-2.3.4.0/bin
lib=/e01/elasticsearch-jdbc-2.3.4.0/lib
DATE=$(date +"%Y-%m-%d %H:%M:%S" -d "5 minutes ago")
#bin=$JDBC_IMPORTER_HOME/bin
#lib=$JDBC_IMPORTER_HOME/lib
echo '{
    "type" : "jdbc",
    "jdbc" : {
                "detect_json":false,
        "elasticsearch.autodiscover":true,
        "elasticsearch.cluster":"elasticsearch",
        "url" : "jdbc:mysql://10.4.32.79:3306/hawkeye_tenant",
        "user" : "tcl",
        "password" : "tcl123",
        "sql" : "select id as _id,tid,skill_id,shop_id,cpro_id,shopids,worksn,addresspath,customername,customertel,bigclass,proscate_id,caller,createtime,updatetime,transfertime,case_code,case_id,engine_id,source_objid,dealstate,dealsource,worktype,sendnum,istransferwork,asktime,askdate,accepttime,departuretime,reachtime,finishtime,replytime,closetime,closereason,planstarttime,planendtime,changenum,create_username,brand_id,probcate_id,dembcate_id,case_level,business_id,phoneext,sms_receipt,cast(extfield as char) as extfield,replysource,isrepair from hw_workorder where updatetime >= \"2016-01-07 00:00:00\"",
        "elasticsearch" : {
             "host" : "10.4.66.56",
             "port" : 9300
        },
        "index" : "hawkeye_tenant4",
        "type" : "hw_workorder4",
        "threadpoolsize" : 8
    }
}' | java \
       -cp "${lib}/*" \
       -Dlog4j.configurationFile=${bin}/log4j2.xml \
       org.xbib.tools.Runner \
       org.xbib.tools.JDBCImporter

这里需要注意的是

1."detect_json":false 不对数据库的extfield的大json进行解析为对象。

2.在canal中获取数据库变更数据对es进行插入时要保持对时间字段格式的一致。

0 0