百度对象存储BOS(Baidu Object Storage)进行冷存储数据备份
来源:互联网 发布:windows vista系统重装 编辑:程序博客网 时间:2024/06/11 10:44
最近有需求就是冷存储数据进行异地灾备,同时为了更多的节省本地的存储成本,维护成本,人力资源等等,选择使用相对更为优惠的百度对象存储来进行备份数据,BOS产品介绍:BOS介绍,为了快速的,批量的上传文件,利用BOS Python SDK开发了一套分布式多任务上传解决方案,本文主要来介绍一下BOS Python SDK的使用方法,为BOS免费做了广告,怎么感谢我!
一、创建虚拟环境
# yum install python-virtualenv# mkvirtualenv bos# workon bos使用virtualenv主要是使用虚拟环境来搭建python开发环境,将不同的python项目进行隔离,避免相关的包的冲突,简单的介绍一下virtualenv的使用:
列出虚拟环境列表:
workon/lsvirtualenv
新建虚拟环境:
mkvirtualenv [虚拟环境名称]
启动/切换虚拟环境:
workon [虚拟环境名称]
删除虚拟环境:
rmvirtualenv [虚拟环境名称]
离开虚拟环境:
deactive
二、安装BOS SDK
2.1 下载bos sdk安装包
URL:https://bce.baidu.com/doc/SDKTool/index.html#Python
wget http://sdk.bce.baidu.com/console-sdk/bce-python-sdk-0.8.8.zip2.2 执行安装脚本
python setup.py install
三、编写配置文件
bos_sample_conf.py:
import loggingimport osimport sysfrom baidubce.bce_client_configuration import BceClientConfigurationfrom baidubce.auth.bce_credentials import BceCredentialsPROXY_HOST = 'localhost:8080' bos_host = "bj.bcebos.com"access_key_id = "a6748c1334a44c2d8af60fcdf098b30d"secret_access_key = "3d7621d35b0c426ea2c0dfdbfca45151"logger = logging.getLogger('baidubce.services.bos.bosclient')fh = logging.FileHandler("sample.log")fh.setLevel(logging.DEBUG)formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')fh.setFormatter(formatter)logger.setLevel(logging.DEBUG)logger.addHandler(fh)config = BceClientConfiguration(credentials=BceCredentials(access_key_id, secret_access_key), endpoint = bos_host)
配置文件指定了上传的host,id和secret,并执行了client的初始化配置config。
四、文件上传
import os,sys,hashlibfrom baidubce import exceptionfrom baidubce.services import bosfrom baidubce.services.bos import canned_aclfrom baidubce.services.bos.bos_client import BosClientimport base64import bos_sample_conf##init a bos clientbos_client = BosClient(bos_sample_conf.config)##init a bucketbucket_name = 'wahaha'if not bos_client.does_bucket_exist(bucket_name): bos_client.create_bucket(bucket_name) print "init bucket:%s success" % bucket_name##upload object from stringobject_key = 'Happy Spring Festival'str = 'this is the test string'bos_client.put_object_from_string(bucket_name,object_key,str)##put object from filefile_name = "/root/baidu_object_storage/test/file_to_be_upload"response = bos_client.put_object_from_file(bucket_name, object_key + ' plus',file_name)print "response.metadata.etag = " + response.metadata.etag##get object meta dataresponse = bos_client.get_object_meta_data(bucket_name, object_key+' plus')print "response object meta data:" print response##list objects in bucketresponse = bos_client.list_objects(bucket_name)for object in response.contents: print 'object.key = ' + object.key##get bucket listresponse = bos_client.list_buckets()for bucket in response.buckets: print "bucket.name = " + bucket.name##get objectprint bos_client.get_object_as_string(bucket_name,object_key)#get unfinished multipart upload taskprint "get unfinished multipart upload task:"for item in bos_client.list_all_multipart_uploads(bucket_name): print 'item.upload_id = ' + item.upload_id#abort unfinished multipart upload taskprint "abort unfinished multipart upload task"for item in bos_client.list_all_multipart_uploads(bucket_name): bos_client.abort_multipart_upload(bucket_name, item.key.encode("utf-8"), upload_id = item.upload_id)response = bos_client.list_multipart_uploads(bucket_name)for item in response.bukcet: print item.name
结果:
五、大文件上传
import osimport sysimport hashlibsys.path.append("../bos")import bos_sample_conffrom baidubce import exceptionfrom baidubce.services import bosfrom baidubce.services.bos import canned_aclfrom baidubce.services.bos.bos_client import BosClientdefault_path = os.path.dirname(os.path.realpath(__file__))#init a bos clientbos_client = BosClient(bos_sample_conf.config)#init a bucketbucket_name = 'wahaha'if not bos_client.does_bucket_exist(bucket_name): bos_client.create_bucket(bucket_name)#init object keyobject_key = 'this is object_key of big file'#upload multipart objectupload_id = bos_client.initiate_multipart_upload(bucket_name,object_key).upload_idprint 'upload_id = ' + upload_idfile_name = default_path + os.path.sep + 'big_file'if os.path.isfile(file_name): print "file_name = %s" % file_nameelse: exit(-1)#set the beginning of multipartleft_size = os.path.getsize(file_name)#set the offset offset = 0part_number = 1part_list = []e_tag_str = ""while left_size > 0: #set each part 50MB print "size left: %dMB" % (left_size/1014/1024) part_size = 50*1024*1024 if left_size < part_size: part_size = left_size response = bos_client.upload_part_from_file( bucket_name,object_key,upload_id,part_number,part_size,file_name,offset) left_size -= part_size offset += part_size part_list.append({ "partNumber":part_number, "eTag":response.metadata.etag }) part_number += 1 e_tag_str += response.metadata.etag print part_number, " ", response.metadata.etagprint "\n"response = bos_client.complete_multipart_upload(bucket_name,object_key,upload_id,part_list)print responsem = hashlib.md5()m.update(e_tag_str)e_tag_str_to_md5 = "-" + m.hexdigest()if e_tag_str_to_md5 == response.etag: print "e_tag match great!!!"else: print "etag does not match, e_tag_str_to_md5 = %s" % e_tag_str_to_md5print "\n"print response.bucketprint response.keyprint response.etagprint response.location结果:
注:
1.对于小文件上传返回的response值的etag值就是小文件的md5值。
2.对于大文件上传,完成分块上传的etag是每一个分块上传返回的etag值相加之后再计算md5值得到的值之前再加个"-"(好抽象)。
Author:忆之独秀
Email:leaguenew@qq.com
注明出处:http://blog.csdn.net/lavorange/article/details/50639849
0 0
- 百度对象存储BOS(Baidu Object Storage)进行冷存储数据备份
- 对象存储(Object-based Storage)概述
- 对象存储(Object-based Storage)概述
- C++永久对象存储 (Persistent Object Storage for C++)
- C++永久对象存储 (Persistent Object Storage for C++)
- 公开付费的非关系数据库:百度对象存储BOS
- Storage Options(数据存储)
- C++永久对象存储 (Persistent Object Storage for C+
- OpenStack对象存储监控(OpenStack Object Storage Monitoring)
- Web Storage 数据存储
- html5 storage数据存储
- 百度富文本编辑器ueditor上传文件到对象存储BOS中
- Android数据存储(External Storage,外部存储)
- Android数据存储-Internal Storage
- Data Storage 数据存储方式
- Android数据存储--External Storage(外部存储)
- Android数据存储(Data Storage)(一)
- Android数据存储(Data Storage)(二)
- jekyll 2 升级 jekyll 3 Dependency Error
- 百思学习笔记02-设置tabbar
- android studio 图片资源的放置位置
- [LeetCode]1. Two Sum
- Hadoop—MapReduce练习(数据去重、数据排序、平均成绩、倒排索引)
- 百度对象存储BOS(Baidu Object Storage)进行冷存储数据备份
- 统计字母,数字,空格的个数
- 【时光碎片】寻找记忆里的春节
- C#百度网盘批量转存助手工具_绿色安全版 百度云盘一键转存小黄人软件原创20161204已测可用
- 查找算法
- 关于php的魔术常量
- CAS部署错误1:Ticket验证异常org.jasig.cas.client.validation.TicketValidationException:
- unityTaidou(一)NGUI之列表的创建与使用
- Ubuntu Kylin 15.04 java环境安装配置