通过ICE接口对hbase进行文件操作

来源：互联网发布：泛型 java 编辑：程序博客网时间：2024/06/05 11:51

最近开发了几个对集群做文件操作的模块，都是基于python（2.7）。下面记录一下对通过ICE接口，写HBASE的模块，贴出部分代码。

1. 关于ICE

ICE是一个中间件，我们可以通过ICE，不考虑server和client分别是什么，进行跨集群的操作。ice已经有很多资料可供学习，以后也会自己总结一个关于ice的文档

2. 依赖包

ICE

3. 源码及解释

模块实现的基本内容就是，连接ice服务，按照一定的json格式，将要操作的文件信息及路径传给ice,定的接口json如下：

zip源：{    "taskId":"asdfsdafdsf2o3r",    "files": ["aaa/bbb/a.txt", "aaa/ccc/b.txt"],    "parent": "/usr/data/input/1/aaa.zip",    "parentType": "zip"}目录源：{    "taskId":"asdfsdafdsf2o3r",    "files": ["aaa/bbb/a.txt", "aaa/ccc/b.txt"],    "parent": "/usr/data/input/1/",    "parentType": "nfs"}入库结果：{    "taskId":"asdfsdafdsf2o3r",    "parent": "/usr/data/input/1/aaa.zip",    "result": [        {"file":"aaa/bbb/a.txt","isSuccess":false,"reason":"putData to hbase failed","length":1234, "costTime":500},        {"file":"aaa/ccc/b.txt","isSuccess":true,"reason":"","length":1234, "costTime":500}]}

实现代码段：

def write_hbase(self):        cfgs = self.configs.configs.app.hbasewriter        for category in self.contents:            try:                self.log_info('begin to write_hbase resource {}', category)                content_info = self.contents[category]                msg = channel.Message(data=channel.MessageData())                uuid_str = uuid.uuid4().hex                content = Info(taskId=uuid_str, files=content_info.files, parent=content_info.parent,parentType=content_info.parentType)                msg.content = content.encode()                msg.data.text = ''                msg.flag = 0                result = channel.sync_call(cfgs.proxy, msg)                result_content = Info().decode(result.content)                if result_content.result:                    for res in result_content.result:                        print('result is {}'.format(res))                        if trans_bool(res.isSuccess):                            self.log_info('write_hbase sucess, file: {}, costTime: {}, length: {}'                                          .format(res.file, res.costTime, res.length))                        else:                            self.log_error('write_hbase fail, file: {}, reason: {}'                                          .format(res.file, res.reason))                else:                    self.log_error('get basic data from interface error:{}', result_content.error)            except:                self.log_exception('error happen in write_hbase')

0 0