COCO Detection Challenge

Introduction

COCO的全称是Common Object in Context, 是 MicroSoft 赞助的CV挑战, 包含 Detections, Captions, Keypoints 三个Challenge, 及相应的数据集.
接下来根据coco 2016介绍它的Detection Challenge.
值得指出的是, coco2016 detection与coco2015一模一样.

Dataset下载

数据集由train/val/tests三部分构成, 总共包含超过20,000张标注图片, 80个类别. 所有的object instance都有bounding box 和segmentation groundtruth, 既可用于detection, 又可用于segmentation.

下载地址 http://mscoco.org/dataset/#download

• 2014 Testing Images在Captioning Challenge中使用
• 2015 Testing Images在Detection和 Keypoint Challenges中使用.
• 2014 Train 和 val Images是所有challenges共用.

所以对于Detection Challenge, 需要用到2014 Training, 2014 Val, 2015 Testing Images 和 2014 Train/Val object instances 共四部分的数据.

Competitors are recommended but not restricted to train their algorithms on COCO 2014 train and val sets. The download page contains links to all COCO 2014 train+val images and associated annotations as well as the 2015 test images. Please specify any and all external data used for training in the “method description” when uploading results to the evaluation server.

官方推荐只使用2014 train/eval训练模型, 但也可以使用其他数据. 提交结果时需要注明.

cocoAPI

官方提供了代码和接口用于读取数据和生成提交结果: https://github.com/pdollar/coco

Annotation Format

Splits与文件信息

下载并解压标注文件:

文件内容的格式相同, 都包含了info, images, annotations, licenses四部分内容, 其中images和annotations是groundtruth的主体.

一个json文件代表一个split, train/val有4个split, test 1个

• image_info_test2014.json
• instances_train2014.json: 82783张图片
• instances_val2014.json: 40504张图片
• instances_minival2014.json: 5000张图片
• instances_valminusminival2014.json: 35504张图片

将val分为minival和valminusminival, 后者可与train一起用于训练.

Object Instance Annotations

COCO 现有三种类型的标注:
1. object instances
2. object keypoints
3. image captions
都存储在Annotations里面. 只有object instances与Detection相关.
instance annotation的格式如下:

• 每个instance 都有segment与bbox标注
• 当一个instance代表单个物体时, iscrowd = 0, segmentation用一个或多个多边形(polygon)表示. 有遮挡时用多个.
• 当一个instance代表一群物体时, iscrowd = 1, segmentation用RLE(Run Length Encoding) 表示.
• segmentation与bbox的都是使用绝对坐标, float类型, 没有normalize到0, 1之间.

结果提交

服务器地址: https://competitions.codalab.org/competitions/5181#learn_the_details-evaluation
需要注册.

To limit overfitting while giving researchers more flexibility to test their system, we have divided the test set into a number of splits, including test-dev, test-standard, and test-challenge. Test-dev is used for debugging and validation experiments and allows for unlimited submission to the evaluation server. Test-standard is used to maintain a public leaderboard that is updated upon submission. Finally, test-challenge is used for the workshop competition; results will be revealed during the workshop at ECCV 2016.

为方便开发调参及防止在test上过拟合/调参, test 数据集分为test-dev与test-std与test-challenge. 前两者现在还可以提交结果. test-dev可以提交很多次(基本可以说不限次数), 用于调试与调参, test-std最多只能提交5次, 用于leadboard排名:

• test-dev(bbox)
  
• test-std(bbox)
  
• test-dev(segm)
  
• test-std(segm)