msgpack[C++]使用笔记 和 msgpack/cPickle性能对比
来源:互联网 发布:java字符串反转函数 编辑:程序博客网 时间:2024/06/05 17:02
python版本msgpack安装:
wget http://pypi.python.org/packages/source/m/msgpack-python/msgpack-python-0.1.9.tar.gz
python2.x setup.py install --prefix=/usr/local/similarlib/
python版本的msgpack灰常好用,速度上比python内置的pickle和cpickle都要快一些,C++版本的使用比较麻烦,下面是本人学习时的一个demo,解析python-msgpack dump的一个复杂字典。
#include <msgpack.hpp>#include <fstream>#include <iostream>using namespace std;template <class T>void msgunpack(const char* binary_file, T& t, char* buff, uint32_t max){msgpack::unpacked msg;ifstream tf_file(binary_file,ios::in|ios::binary|ios::ate);uint32_t size = tf_file.tellg();tf_file.seekg(0, ios::beg);tf_file.read(buff, size);tf_file.close();msgpack::unpack(&msg, buff, size);msg.get().convert(&t);}typedef map<uint32_t, uint32_t> WordsMap;typedef map<uint32_t, WordsMap> FieldsMap;typedef map<uint64_t, FieldsMap> DocsMap;int main(int argc, char** argv){uint32_t MAX_BUFF = 1024*1024*100; //100MBchar* BUFF = new char[MAX_BUFF];DocsMap docsMap;msgpack::unpacked msg;msgunpack("/data/wikidoc/tf_dict_for_nodes/1-1000", docsMap, BUFF, MAX_BUFF);// msg.get().convert(&docsMap);cout << docsMap.size() << endl; delete[] BUFF;}
参考: http://wiki.msgpack.org/pages/viewpage.action?pageId=1081387#QuickStartforC%2B%2B-ImplementationStatus
下面是本人自己封装的一个msgpack接口头文件mymsgpack.h
#ifndef MY_MSGPACK_H#ifndef MY_MSGPACK_H#define MY_MSGPACK_H#include <fstream>#include <msgpack.hpp>using namespace std;template <class T>void load_from_file(const char* binary_file, T& t) { ifstream binaryFstream(binary_file,ios::in|ios::binary|ios::ate); uint32_t size = binaryFstream.tellg(); char* buff = new char[size]; binaryFstream.seekg(0, ios::beg); binaryFstream.read(buff, size); binaryFstream.close(); msgpack::unpacked msg; msgpack::unpack(&msg, buff, size); msg.get().convert(&t); delete[] buff;}template <class T>void load_from_str(const char* binary_str, int len, T& t) { msgpack::unpacked msg; msgpack::unpack(&msg, binary_str, len); msg.get().convert(&t);}template <class T>void dump_to_file(T& t, const char* dump_file) {msgpack::sbuffer sbuf;msgpack::pack(sbuf, t);ofstream dumpFstream(dump_file, ios::out|ios::binary|ios::trunc);dumpFstream.write(sbuf.data(), sbuf.size());dumpFstream.close();}template <class T>void dump_to_str(T& t, char** dump_str, int& len) { //外部释放*dump_strmsgpack::sbuffer sbuf;msgpack::pack(sbuf, t);len = sbuf.size();*dump_str = (char*)malloc(sbuf.size());memcpy(*dump_str, sbuf.data(), sbuf.size());}#endif
msgpack编译通过,链接不上的问题 undefined reference to `__sync_sub_and_fetch_4'
在x84_64机器上正常,在32bit机器上出现上述问题
[xudongsong@BigServerU-4 msgpack-0.5.7]$ cat /etc/issueCentOS release 5.4 (Final)Kernel \r on an \m[xudongsong@BigServerU-4 msgpack-0.5.7]$ file /sbin/init/sbin/init: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped
./configure不报错,但是查看config.log显示有错误,程序链接msgpack的库时也报错
原因:gcc不能识别CPU体系,需要手动指明
[xudongsong@BigServerU-4 msgpack-0.5.7]$ CFLAGS="-march=pentium -mtune=pentium" ./configure --prefix=/home/xudongsong/msgpack_static --enable-static=yes --enable-shared=no
make, make install
[xudongsong@BigServerU-4 jobs]$ g++ job_calc_weight.cpp -o job_calc_weight -I/home/xudongsong/msgpack_static/include/ -L/home/xudongsong/msgpack_static/lib/ -lmsgpack
通过!
下面是msgpack和cPickle进行性能pk的demo程序(不比较pickle,是因为它比cPickle更慢,《Python cook book》里面有说明):
mport sys,time,msgpack,pickle,cPickle,randomtest_list = []i = 0while i<100000:test_list = random.randrange(1,100000)i += 1print "common len(serialize) = %s"%len(cPickle.dumps(test_list,0))print "compress len(serialize) = %s"%len(cPickle.dumps(test_list,1))#------------------------------------------------------------------------results = {}time_start = time.time()for i in range(1,1000000): results[i] = cPickle.dumps(test_list,1)time_mid_1 = time.time()print "cPickle dumps eats %s s"%str(time_mid_1-time_start)for i in range(1,1000000):cPickle.loads(results[i])time_mid_2 = time.time()print "cPickle loads eats %s s"%str(time_mid_2-time_mid_1)#------------------------------------------------------------------------results = {}time_start = time.time()for i in range(1,1000000):results[i] = msgpack.dumps(test_list)time_mid_1 = time.time()print "msgpack pack eats %s s"%str(time_mid_1-time_start)for i in range(1,1000000):msgpack.loads(results[i])time_mid_2 = time.time()print "msgpack unpack eats %s s"%str(time_mid_2-time_mid_1)
- msgpack[C++]使用笔记 和 msgpack/cPickle性能对比
- msgpack[C++]使用笔记 和 msgpack/cPickle性能对比
- msgpack[C++]使用笔记 和 msgpack/cPickle性能对比
- msgpack
- msgpack使用
- php implode/explode, serialize, json, msgpack 性能对比
- msgpack的使用
- linux使用msgpack及测试
- MSGPACK和PROTOBUF的故事(MSGPACK明显生产力不足)
- msgpack 原理
- msgpack用法
- 安装msgpack
- thrift, Protocol Buffers and MsgPack 的序列化对比
- thrift, Protocol Buffers and MsgPack 的序列化对比
- 使用MsgPack配合Qt应用程序和node.js应用程序的数据传输
- jsonsmart,msgpack,protocalbuffer谁更强?
- massage pack msgpack
- msgpack---序列化
- QML实现-按钮点击变色功能
- Flex连接数据库的3种方式
- 正确地做事与做正确的事同样重要
- 关于Ext.onReady的三种调用方法
- JAVA中的反射机制详解
- msgpack[C++]使用笔记 和 msgpack/cPickle性能对比
- 一个好的测试人员该具备什么素质?
- 在线安装p7zip-full地址
- 下拉工具栏的设计
- `process_cache_miss' 错误
- Linux下ch341USB转串口芯片的问题
- jsoup解析html
- Collections的用法
- jpeg组包