阿里 离线数据同步工具 DataX 初试

DataX ： 一个异构数据源离线同步框架，通过插件体系完成数据同步过程。reader插件用于读入，writer插件用于写出，中间的framework可以定义transform插件完成数据转化的需要。

Sqoop 只支持关系型数据库与HDFS/Hive 之间的数据同步， DataX 则更为丰富。

目前支持的数据源有：https://github.com/alibaba/DataX/wiki/DataX-all-data-channels

使用：

$ tar zxvf datax.tar.gz$ sudo chmod -R 755 {YOUR_DATAX_HOME}$ cd  {YOUR_DATAX_HOME}/bin$ python datax.py ../job/job.json

json配置例子（Mongo > HDFS/Hive）：

mongotest.json

{    "job": {        "setting": {            "speed": {                "channel": "2"            }        },        "content": [{                "reader": {                    "name": "mongodbreader",                    "parameter": {                        "address": [""],                        "userName": "",                        "userPassword": "",                        "dbName": "",                        "collectionName": "",                        "column": [{                                "name": "cityid",                                "type": "string"                            }, {                                "name": "searchstr",                                "type": "string"                            }, {                                "name": "pv",                                "type": "string"                            }                        ]                    }                },                "writer": {                    "name": "hdfswriter",                    "parameter": {                        "column": [{                                "name": "cityid",                                "type": "string"                            }, {                                "name": "searchstr",                                "type": "int"                            }, {                                "name": "pv",                                "type": "int"                            }                        ],                        "defaultFS": "hdfs://*",                        "fieldDelimiter": "\t",                        "fileName": "mongotest",                        "fileType": "text",                        "path": "/user/hive/warehouse/temp.db/mongotest",                        "writeMode": "append"                    }                }            }        ]    }}

同步过程：

1. create Hive table temp.mongotest
2. python {DATAX_HOME}/bin/datax.py ../mongotest.json