[训练测试过程记录]SSD:Single Shot Detector 用于场景文字检测（举例数据集：COCO-Text）

介绍用SSD模型进行场景文字检测。举例数据集：COCO-Text。

编译部分：

1.使用cuda8编译时出错

/usr/include/boost/property_tree/detail/json_parser_read.hpp:257:264: error: ‘type name’ declared as function returning an array

解决方法：因为gcc版本过低，升级到5.3即可解决.

sudo add-apt-repository ppa:ubuntu-toolchain-r/test  sudo apt-get update     sudo apt-get install software-properties-common    sudo apt-get install gcc-5 g++-5    cd /usr/bin    sudo rm gcc    sudo ln -s gcc-5 gcc    sudo rm g++    sudo ln -s g++-5 g++  

重新编译即可解决

2. make: *** [.build_release/lib/libcaffe.so.1.0.0-rc3] 错误 

解决方法：

sudo apt-get install libopenblas-dev

同样，安装后，再重新编译即可解决

数据集准备部分：

使用coco-text数据集

1.将coco-text数据集格式化为pascal_voc的数据集格式，格式方法详见博客：[训练测试过程记录]Text-Detection-with-FRCN中的第二部分：准备数据集，这里不再赘述。

2.将formatted_dataset更名为VOC2007，并放入文件夹$home/data/VOCdevkit下面。

3.创建Imdb格式的数据：

cd $CAFFE_ROOT# Create the trainval.txt, test.txt, and test_name_size.txt in data/VOC0712/./data/VOC0712/create_list.sh# You can modify the parameters in create_data.sh if needed.# It will create lmdb files for trainval and test with encoded original image:#   - $HOME/data/VOCdevkit/VOC0712/lmdb/VOC0712_trainval_lmdb#   - $HOME/data/VOCdevkit/VOC0712/lmdb/VOC0712_test_lmdb# and make soft links at examples/VOC0712/./data/VOC0712/create_data.sh

注意：

1.修改create_list.sh和create_data.sh下面的数据集路径

create_list.sh:

#root_dir=$HOME/data/VOCdevkit/root_dir="改为自己的数据集目录'sub_dir=ImageSets/Mainbash_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"for dataset in trainval testdo  dst_file=$bash_dir/$dataset.txt  if [ -f $dst_file ]  then    rm -f $dst_file  fi  #for name in VOC2007 VOC2012  #这里只有VOC2007文件夹  for name in VOC2007  do    if [[ $dataset == "test" && $name == "VOC2012" ]]    then      continue

create_data.sh：

cur_dir=$(cd $( dirname ${BASH_SOURCE[0]} ) && pwd )root_dir=$cur_dir/../..cd $root_dirredo=1#data_root_dir="$HOME/data/VOCdevkit"data_root_dir="改为自己的数据集目录"dataset_name="VOC0712"mapfile="$root_dir/data/$dataset_name/labelmap_voc.prototxt"anno_type="detection"

2.修改labelmap_voc.prototxt下的数据集类别

由于这里只有背景和text两类，因此改为：

item {  name: "none_of_the_above"  label: 0  display_name: "background"}item {  name: "text"  label: 1  display_name: "text"}

训练部分：

# It will create model definition files and save snapshot models in:#   - $CAFFE_ROOT/models/VGGNet/VOC0712/SSD_300x300/# and job file, log file, and the python script in:#   - $CAFFE_ROOT/jobs/VGGNet/VOC0712/SSD_300x300/# and save temporary evaluation results in:#   - $HOME/data/VOCdevkit/results/VOC2007/SSD_300x300/# It should reach 77.* mAP at 120k iterations.python examples/ssd/ssd_pascal.py

问题１：num_test_image的数目不对

解决方法：需要将4952换成测试图片的数量，例如coco-text的测试集数目为840。

#Evaluate on whole test set.#num_test_image = 4952num_test_image = 840test_batch_size = 8# Ideally test_batch_size should be divisible by num_test_image,# otherwise mAP will be slightly off the true value.test_iter = int(math.ceil(float(num_test_image) / test_batch_size))

问题２：loss = nan

由于是场景文字数据集的原因，初始迭代产生的loss特别大，我自己训练是到了iteration 40的时候，就开始变成Loss = nan了。

原因：梯度爆炸。梯度变得非常大，使得学习过程难以继续。

一般措施：减小solver.prototxt的base_lr，至少减小一个数量级。如果有多个loss layer，需要找出哪个损失层导致了梯度爆炸，并在train_val.prototxt中减小该层的loss_weight，而非是减小通用的base_lr。参考：使用caffe训练时Loss变为nan的原因

解决方法：

将base_lr变为原来的10倍。在/examples/ssd/ssd_pascal.py的第229行和232行处进行修改，将0.004改为0.0004和将0.00004改为0.000004。

# If true, use batch norm for all newly added layers.# Currently only the non batch norm version has been tested.use_batchnorm = Falselr_mult = 1# Use different initial learning rate.if use_batchnorm:　  #base_lr = 0.0004    base_lr = 0.00004else:    # A learning rate for batch_size = 1, num_gpus = 1.    #base_lr = 0.00004    base_lr = 0.000004

PS:降低学习率可能会带来loss收敛速度很慢的问题。之前想过更换训练模型，也就是将官方给点的pretrained model：fully convolutional reduced (atrous) VGGNet换成训练好了的SSD300*模型。但是由于维度不一样，将num_classes的维度由21换到了2，所以只能使用官方给的pretrained model。目前除了降低base_lr，还没有想到其他更好的办法。

问题３：OpenCV Error: Assertion failed

OpenCV Error: Assertion failed ((scn == 3 || scn == 4) && (depth == CV_8U || depth == CV_32F)) in ipp_cvtColor, file /home/user1/opencv-3.1.0/modules/imgproc/src/color.cpp, line 7646terminate called after throwing an instance of 'cv::Exception'what(): /home/user1/opencv-3.1.0/modules/imgproc/src/color.cpp:7646: error: (-215) (scn == 3 || scn == 4) && (depth == CV_8U || depth == CV_32F) in function ipp_cvtColor*** Aborted at 1482480286 (unix time) try "date -d @1482480286" if you are using GNU date ***PC: @ 0x7f7e541abcc9 (unknown)*** SIGABRT (@0x3e900004df8) received by PID 19960 (TID 0x7f7e227fd700) from PID 19960; stack trace: ***@ 0x7f7e541abd40 (unknown)@ 0x7f7e541abcc9 (unknown)@ 0x7f7e541af0d8 (unknown)@ 0x7f7e54f61535 (unknown)@ 0x7f7e54f5f6d6 (unknown)@ 0x7f7e54f5f703 (unknown)@ 0x7f7e54f5f922 (unknown)@ 0x7f7e4d12fca0 cv::error()@ 0x7f7e4d12fe20 cv::error()@ 0x7f7e4b574c89 cv::ipp_cvtColor()@ 0x7f7e4b57e4d4 cv::cvtColor()@ 0x7f7e5600758a caffe::AdjustSaturation()@ 0x7f7e5600c77a caffe::RandomSaturation()@ 0x7f7e5600ce96 caffe::ApplyDistort()@ 0x7f7e561dfeac caffe::DataTransformer<>::DistortImage()@ 0x7f7e561c7beb caffe::AnnotatedDataLayer<>::load_batch()@ 0x7f7e560abc29 caffe::BasePrefetchingDataLayer<>::InternalThreadEntry()@ 0x7f7e55ff39f0 caffe::InternalThread::entry()@ 0x7f7e4abf9a4a (unknown)@ 0x7f7e4678a182 start_thread@ 0x7f7e5426f47d (unknown)@ 0x0 (unknown)Aborted (core dumped)

解决方法：在/examples/ssd/ssd_pascal.py的第175行的train_transform_param中添加'force_color':True,

参考：OpenCV Error: Assertion failed #353

train_transform_param = {        'mirror': True,        'mean_value': [104, 117, 123],　　     #added'force_color': True,        'resize_param': {                'prob': 1,                'resize_mode': P.Resize.WARP,                'height': resize_height,                'width': resize_width,                'interp_mode': [                        P.Resize.LINEAR,                        P.Resize.AREA,                        P.Resize.NEAREST,                        P.Resize.CUBIC,                        P.Resize.LANCZOS4,                        ],                },

问题４：Check failed: mean_values_.size() == 1

F1203 16:07:24.865304 12717 data_transformer.cpp:621] Check failed: mean_values_.size() == 1 || mean_values_.size() == img_channels Specify either 1 mean_value or as many as channels: 1*** Check failure stack trace: ***    @     0x7f6168187daa  (unknown)    @     0x7f6168187ce4  (unknown)    @     0x7f61681876e6  (unknown)    @     0x7f616818a687  (unknown)    @     0x7f61689df73d  caffe::DataTransformer<>::Transform()    @     0x7f61689e0993  caffe::DataTransformer<>::Transform()    @     0x7f61689ebcdb  caffe::DataTransformer<>::Transform()    @     0x7f61689ebd98  caffe::DataTransformer<>::Transform()    @     0x7f61689ebe3e  caffe::DataTransformer<>::Transform()    @     0x7f616887005b  caffe::AnnotatedDataLayer<>::load_batch()    @     0x7f616884f6dc  caffe::BasePrefetchingDataLayer<>::InternalThreadEntry()    @     0x7f61689a1445  caffe::InternalThread::entry()    @     0x7f615e23ba4a  (unknown)    @     0x7f615729c184  start_thread    @     0x7f6166aabbed  (unknown)    @              (nil)  (unknown)

解决方法：在/examples/ssd/ssd_pascal.py的第213行的test_transform_param中添加'force_color':True,

参考：Training error for face detection training!

test_transform_param = {        'mean_value': [104, 117, 123],        'force_color': True,        'resize_param': {                'prob': 1,                'resize_mode': P.Resize.WARP,                'height': resize_height,                'width': resize_width,                'interp_mode': [P.Resize.LINEAR],                },        }

问题５：status == CUDNN_STATUS_SUCCESS

F0616 16:54:55.034394 3070141376 cudnn_conv_layer.cpp:53] Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0) CUDNN_STATUS_INTERNAL_ERROR

原因：内存不够

解决方法：减小batch_size，例如我在这里将训练的batch_size由32减小到了8，修改处为在/examples/ssd/ssd_pascal.py的第338行和第339行。

#Divide the mini-batch to different GPUs.#batch_size = 32#accum_batch_size = 32batch_size = 8accum_batch_size = 8iter_size = accum_batch_size / batch_sizesolver_mode = P.Solver.CPUdevice_id = 0

问题6：Check failed: label_to_name_.find(label) !=lable_to_name_.name() Cannot find label: 2 in the label map

F1027 detection_output_layer.cu:143] Check failed: label_to_name_.find(label) !=lable_to_name_.name() Cannot find label: 2 in the label map

这是由于将类别由21类改为2类造成的。

解决方法：

1.将examples/ssd/ssd_pascal.py中的269行中的num_classes ，由21改为2。

# MultiBoxLoss parameters.# num_classes = 21num_classes = 2share_location = Truebackground_label_id=0train_on_diff_gt = Truenormalization_mode = P.Loss.VALIDcode_type = P.PriorBox.CENTER_SIZEignore_cross_boundary_bbox = Falsemining_type = P.MultiBoxLoss.MAX_NEGATIVEneg_pos_ratio = 3.

2.将examples/ssd/score_ssd_pascal.py中的277行中的num_classes ，由21改为2。

# MultiBoxLoss parameters.# num_classes = 21num_classes = 2share_location = Truebackground_label_id=0train_on_diff_gt = Truenormalization_mode = P.Loss.VALIDcode_type = P.PriorBox.CENTER_SIZEignore_cross_boundary_bbox = Falsemining_type = P.MultiBoxLoss.MAX_NEGATIVEneg_pos_ratio = 3.

在py文件中改过之后，以下目录中的train.prototxt，test.prototxt，deploy.prototxt中的num_classes也会随之改变：

caffe/jobs/VGGNet/VOC0712/SSD_300x300
caffe/jobs/VGGNet/VOC0712/SSD_300x300_score


caffe/models/VGGNet/VOC0712/SSD_300x300
caffe/models/VGGNet/VOC0712/SSD_300x300_score