Large Object Support大对象支持

Large Object Support大对象支持

Overview概述

Swift has a limit on the size of a single uploadedobject; by default this is 5GB. However, the download size of a single objectis virtually unlimited with the concept of segmentation. Segments of the largerobject are uploaded and a special manifest file is created that, whendownloaded, sends all the segments concatenated as a single object. This alsooffers much greater upload speed with the possibility of parallel uploads ofthe segments.

Siwft对于单个上传对象有体积的限制；默认是5GB。不过由于使用了分割的概念，单个对象的下载大小几乎是没有限制的。对于更大的对象进行分割然后上传并且会创建一个特殊的描述文件，当下载该对象的时候，把所有的分割联接为一个单个对象来发送。这使得并行上传分割成为可能，因此也提供了更快的上传速度。

Dynamic Large Objects

Using swift

The quickest way to try out this feature is use the swift Swift Tool includedwith the python-swiftclientlibrary. You can use the -S option to specifythe segment size to use when splitting a large file. For example:

尝试这一特性的最快捷的方式是使用swift自带的Swift Tool。你可以使用-S选项来描述在分割大文件的时候使用的分卷大小。例如：

swiftupload test_container -S 1073741824 large_file

This would split the large_file into 1G segments andbegin uploading those segments in parallel. Once all the segments have beenuploaded, swift willthen create the manifest file so the segments can be downloaded as one.

这个会把large_file分割为1G的分卷并且开始并行地上传这些分卷。一旦所有的分卷上传完毕，swift将会创建描述文件，这样这些分卷可以作为一个对象来下载。

So now, the following swift command would download the entirelarge object:

所以现在，使用以下swift命令可以下载整个大对象：

swiftdownload test_container large_file

swift command uses astrict convention for its segmented object support. In the above example itwill upload all the segments into a second container namedtest_container_segments. These segments will have names likelarge_file/1290206778.25/21474836480/00000000, large_file/1290206778.25/21474836480/00000001,etc.

swift使用一个严格的约定对于它的分卷对象支持。在上面的例子中，它将会上传所有的分卷到一个名为test_container_segments的附加容器。这些分卷的名称类似于 large_file/1290206778.25/21474836480/00000000,large_file/1290206778.25/21474836480/00000001等。

The main benefit for using a separate container is thatthe main container listings will not be polluted with all the segment names.The reason for using the segment name format of<name>/<timestamp>/<size>/<segment> is so that an uploadof a new file with the same name won’t overwrite the contents of the firstuntil the last moment when the manifest file is updated.

使用一个独立的容器的主要好处是主容器列表将不会被所有的分卷名字污染。使用<name>/<timestamp>/<size>/<segment>分卷名称格式的理由是当上传一个相同名称的新文件时将不会重写先前文件的内容直到最后描述文件被上传的时候。

swift will manage thesesegment files for you, deleting old segments on deletes and overwrites, etc.You can override this behavior with the --leave-segments option if desired;this is useful if you want to have multiple versions of the same large objectavailable.

swift将会为你管理这些分卷文件，使用删除和重写等方法来删除旧的分卷。若需要，你可以用--leave-segments选项重写这一行为；如果你想要同个大对象的多个版本可用这将非常有用。

Direct API直接的API

You can also work with the segments and manifestsdirectly with HTTP requests instead of having swiftdo that for you. You can just upload thesegments like you would any other object and the manifest is just a zero-byte(not enforced) file with an extra X-Object-Manifest header.

你也可以直接用HTTP请求代替swift工具来使用分卷和描述文件。你可以只上传分卷，在带有一个额外的X-Object-Manifest头部中指明任何其他的对象和描述文件只是一个0字节的文件。

All the object segments need to be in the same container,have a common object name prefix, and sort in the order in which they should beconcatenated. Object names are sorted lexicographically as UTF-8 byte strings.They don’t have to be in the same container as the manifest file will be, whichis useful to keep container listings clean as explained above with swift.

所有的对象分卷需要在同一个容器内，有一个相同的对象名称前缀，并且它们的名称按照连结的顺序排序。它们不用和描述文件在同一个容器下，这与上面解释swift组件中一样有助于保持容器列表的干净。

The manifest file is simply a zero-byte (not enforced)file with the extra X-Object-Manifest:<container>/<prefix> header, where <container> is the container the object segments are in and<prefix> is thecommon prefix for all the segments.

描述文件仅是一个带有额外X-Objetc-Manifest的0字节文件：<container>/<prefix>头部，其中<container>是指对象分卷所在的容器，<prefix>是所有分卷的通用前缀。

It is best to upload all the segments first and thencreate or update the manifest. In this way, the full object won’t be availablefor downloading until the upload is complete. Also, you can upload a new set ofsegments to a second location and then update the manifest to point to this newlocation. During the upload of the new segments, the original manifest willstill be available to download the first set of segments.

最好先上传所有的分卷并且然后创建或升级描述文件。在这种方式下，完整的对象的下载直到上传完成才可用。此外，你可以上传一个新的分卷集到新的位置，然后上传描述文件来指出这一新位置。在上传这些新分卷的时候，原始的描述文件将仍然可用来下载第一个分卷集合。

Note

 The manifest file should have no content. However, this is not enforced.If the manifest path itself conforms to container/prefix specified inX-Object-Manifest, and if manifest has some content/data in it, it would alsobe considered as segment and manifest’s content will be part of theconcatenated GET response. The order of concatenation follows the usual DLOlogic which is - the order of concatenation adheres to order returned whensegment names are sorted.

Here’s an example using curl with tiny 1-byte segments:

这里有一个使用curl对1字节的小分卷的例子：

# First, upload the segments

curl -X PUT -H 'X-Auth-Token:<token>' \

   http://<storage_url>/container/myobject/1 --data-binary '1'

curl -X PUT -H 'X-Auth-Token:<token>' \

   http://<storage_url>/container/myobject/2 --data-binary '2'

curl -X PUT -H 'X-Auth-Token:<token>' \

   http://<storage_url>/container/myobject/3 --data-binary '3'

 

# Next, create the manifest file

curl -X PUT -H 'X-Auth-Token:<token>' \

   -H 'X-Object-Manifest: container/myobject/' \

   http://<storage_url>/container/myobject --data-binary ''

# And now we can download the segmentsas a single object

curl -H 'X-Auth-Token: <token>' \

   http://<storage_url>/container/myobject

Static Large Objects

Direct API

SLO support centers around the user generated manifestfile. After the user has uploaded the segments into their account a manifestfile needs to be built and uploaded. All object segments, except the last, mustbe above 1 MB (by default) in size. Please see the SLO docs for Static LargeObjects further details.

Additional Notes其他注意事项

• With a GET or HEAD of a manifest file, the X-Object-Manifest: <container>/<prefix> header will be returned with the concatenated object so you can tell where it’s getting its segments from.

带有GET或者HEAD的描述文件，X-Object-Manifest:<container>/<prefix>头部将会返回被连结的对象，这样你可以辨别它从哪里获得它的分卷。

• The response’s Content-Length for a GET or HEAD on the manifest file will be the sum of all the segments in the <container>/<prefix> listing, dynamically. So, uploading additional segments after the manifest is created will cause the concatenated object to be that much larger; there’s no need to recreate the manifest file.

在描述文件上的GET或HEAD请求的Content-Length是所有在<container>/<prefix>列表中的分卷的动态总和。因此，在创建了描述文件之后上传额外的分卷将会导致连结对象变得更大；没有需要去重新创建描述文件。

• The response’s Content-Type for a GET or HEAD on the manifest will be the same as the Content-Type set during the PUT request that created the manifest. You can easily change the Content-Type by reissuing the PUT.

GET或HEAD描述文件的请求返回的 Content-Type和在创建描述文件的PUT请求中的Content-Type设置一样。你可以通过重新发出PUT请求来轻松地修改Content-Type

• The response’s ETag for a GET or HEAD on the manifest file will be the MD5 sum of the concatenated string of ETags for each of the segments in the manifest (for DLO, from the listing<container>/<prefix>). Usually in Swift the ETag is the MD5 sum of the contents of the object, and that holds true for each segment independently. But it’s not meaningful to generate such an ETag for the manifest itself so this method was chosen to at least offer change detection.

GET或HEAD描述文件的请求的ETag是<container>/<prefix>所列的连结每个分卷的ETags的字符串的MD5值的动态总和。在Swift中Etag常常是对象内容的MD5值总和，并且适用于每个分卷。但是，为描述文件本身来创建这样一个Etag是不可行的，因此这个方法被选择来至少提供变更检测。

Note

 If you are using the container sync feature you will need to ensure bothyour manifest file and your segment files are synced if they happen to be indifferent containers.

如果你选择了容器同步的特性，你将需要来确保你的描述文件和你的分卷文件被同步若它们在不同的容器中。

History发展史

Dynamic large object support has gone through variousiterations before settling on this implementation.

大对象的支持在设为现在这种实现方式前已经经历了各种反复修改。

The primary factor driving the limitation of object sizein swift is maintaining balance among the partitions of the ring. To maintainan even dispersion of disk usage throughout the cluster the obvious storagepattern was to simply split larger objects into smaller segments, which couldthen be glued together during a read.

在swift中驱使限制对象大小的主要因素是维持ring中的partiton间的平衡。为了在集群中维持磁盘使用的平坦散布，一种显而易见的方式是简单地将较大的对象分割到更小的分卷，在读取时分卷可以被粘连在一起。

Before the introduction of large object support someapplications were already splitting their uploads into segments andre-assembling them on the client side after retrieving the individual pieces.This design allowed the client to support backup and archiving of large datasets, but was also frequently employed to improve performance or reduce errorsdue to network interruption. The major disadvantage of this method is thatknowledge of the original partitioning scheme is required to properlyreassemble the object, which is not practical for some use cases, such as CDNorigination.

在介绍大型对象支持之前，一些应用已经将它们的上载对象分割为分卷并且在检索出这些独立块之后在客户端上重新装配它们。这一设计允许客户端支持备份和将大的数据集存档，但也频繁地使用来提升性能或减少由于网络中断引发的错误。这一方法的主要缺点是需要初始的分割组合的知识来合适地将对象重新装配，对于一些使用场景来说是不切实际的，诸如CDN源。

In order to eliminate any barrier to entry for clientswanting to store objects larger than 5GB, initially we also prototyped fullytransparent support for large object uploads. A fully transparentimplementation would support a larger max size by automatically splittingobjects into segments during upload within the proxy without any changes to theclient API. All segments were completely hidden from the client API.

为了解决客户想要存储大于5GB的对象障碍，最初的我们原型化完全透明的对于上传大对象的支持。一个完全透明的实现可以在上传时通过自动地将对象分割为分卷在代理内对于客户端API没有任何变化来支持更大的最大分卷大小。

This solution introduced a number of challenging failureconditions into the cluster, wouldn’t provide the client with any option to doparallel uploads, and had no basis for a resume feature. The transparentimplementation was deemed just too complex for the benefit.

这一解决方案引入了大量的有挑战性的失败条件到集群中，不会提供客户端任何选项来进行并行上传，而且没有把重新开始特性作为基础。这一透明实现被认为对于好处来说是太复杂了。

The current “user manifest” design was chosen in order toprovide a transparent download of large objects to the client and still providethe uploading client a clean API to support segmented uploads.

当前的“用户描述”设计被挑选出来为了提供大型对象到客户的透明下载并且仍然对上载客户端提供了干净的API来支持分卷上载。

To meet an many use cases as possible swift supports twotypes of large object manifests. Dynamic and static large object manifests bothsupport the same idea of allowing the user to upload many segments to be laterdownloaded as a single file.

Dynamic large objects rely on a container listing toprovide the manifest. This has the advantage of allowing the user toadd/removes segments from the manifest at any time. It has the disadvantage ofrelying on eventually consistent container listings. All three copies of thecontainer dbs must be updated for a complete list to be guaranteed. Also, allsegments must be in a single container, which can limit concurrent uploadspeed.

Static large objects rely on a user provided manifestfile. A user can upload objects into multiple containers and then referencethose objects (segments) in a self generated manifest file. Future GETs to thatfile will download the concatenation of the specified segments. This has theadvantage of being able to immediately download the complete object once themanifest has been successfully PUT. Being able to upload segments into separatecontainers also improves concurrent upload speed. It has the disadvantage thatthe manifest is finalized once PUT. Any changes to it means it has to bereplaced.

Between these two methods the user has great flexibilityin how (s)he chooses to upload and retrieve large objects to swift. Swift doesnot, however, stop the user from harming themselves. In both cases the segmentsare deletable by the user at any time. If a segment was deleted by mistake, adynamic large object, having no way of knowing it was ever there, would happilyignore the deleted file and the user will get an incomplete file. A staticlarge object would, when failing to retrieve the object specified in themanifest, drop the connection and the user would receive partial results.