在caffe中用python添加confusion matrix层

confusion matrix(混淆矩阵)在分类问题中能比单纯的准确率更全面地反应模型的好坏，本文主要目的是在caffe中用python添加confusion matrix层。

Step1：准备confusion matrix的python实现代码

完整代码如下：

import caffeimport jsonimport numpy as npimport sysimport sklearn.metricsclass PythonConfMat(caffe.Layer):    """    Compute the Accuracy with a Python Layer    """    def setup(self, bottom, top):        # check input pair        if len(bottom) != 2:            raise Exception("Need two inputs.")        self.num_labels = bottom[0].channels        params = json.loads(self.param_str)        self.test_iter = params['test_iter']        self.conf_matrix = np.zeros((self.num_labels, self.num_labels))        self.current_iter = 0    def reshape(self, bottom, top):        # bottom[0] are the net's outputs        # bottom[1] are the ground truth labels        # Net outputs and labels must have the same number of elements        if bottom[0].num != bottom[1].num:            raise Exception("Inputs must have the same number of elements.")        # accuracy output is scalar        top[0].reshape(1)    def forward(self, bottom, top):        self.current_iter += 1        # predicted outputs        pred = np.argmax(bottom[0].data, axis=1)        accuracy = np.sum(pred == bottom[1].data).astype(np.float32) / bottom[0].num        top[0].data[...] = accuracy        # compute confusion matrix        self.conf_matrix += sklearn.metrics.confusion_matrix(bottom[1].data, pred, labels=range(self.num_labels))        if self.current_iter == self.test_iter:            self.current_iter = 0            sys.stdout.write('\nCAUTION!! test_iter = %i. Make sure this is the correct value' % self.test_iter)            sys.stdout.write('\n"param_str: \'{"test_iter":%i}\'" has been set in the definition of the PythonLayer' % self.test_iter)            sys.stdout.write('\n\nConfusion Matrix')            sys.stdout.write('\t'*(self.num_labels-2)+'| Accuracy')            sys.stdout.write('\n'+'-'*8*(self.num_labels+1))            sys.stdout.write('\n')            for i in range(len(self.conf_matrix)):                for j in range(len(self.conf_matrix[i])):                    sys.stdout.write(str(self.conf_matrix[i][j].astype(np.int))+'\t')                sys.stdout.write('| %3.2f %%' % (self.conf_matrix[i][i]*100 / self.conf_matrix[i].sum()))                sys.stdout.write('\n')            sys.stdout.write('Number of test samples: %i \n\n' % self.conf_matrix.sum())            # reset conf_matrix for next test phase            self.conf_matrix = np.zeros((self.num_labels, self.num_labels))    def backward(self, top, propagate_down, bottom):        pass

注意：此python脚本应放置在$caffe_root/python目录下。

Step2：编译用python新添加的层

若caffe还没有编译，可以打开makefile.config中 WITH_PYTHON_LAYER=1的开关，然后make编译，再然后make pycaffe编译。
若caffe已经编译好了，则需要先make clean，然后用WITH_PYTHON_LAYER=1 make all -j&& make pycaffe编译caffe源码。

Step3：在prototxt中使用confusion matrix layer

在测试网络的prototxt中添加把混淆矩阵层添加进去，列子如下

layer {  type: 'Python'  name: 'py_accuracy'  top: 'py_accuracy'  bottom: 'fc101'  bottom: 'label'  python_param {    # the module name -- usually the filename -- that needs to be in $PYTHONPATH    module: 'python_confmat'    # the layer name -- the class name in the module    layer: 'PythonConfMat'    # this is the number of test iterations, it must be the same as defined in the solver.    param_str: '{"test_iter":100}'  }  include {    phase: TEST  }}

注意：test_iter乘以batch size应cover所有测试集样本。

Step4：使用/build/tools/caffe test工具计算测试集的confusion matrix

这个需要提前把测试机生成lmdb文件。生成后用如下姿势使用

CAFFE_ROOT=/home/***/caffe                                                                         MODEL_ROOT=/home/***/work/resnet_18_age_7                                                $CAFFE_ROOT/build/tools/caffe test --model=$MODEL_ROOT/ResNet-18-test.prototxt \                            --weights ./model_lrstep6/resnet18_age_finetue_iter_43883.caffemodel \                               --iterations 1392 --gpu=1 

caffe test中不需要solver.prototxt，所以在命令中需要指定--iterations. 这里的iterations需要与PythonConfMat层中的test_iter相同。

注意：调用test命令可能出现import error：no module named python_confmat
解决方案：终端输入export PYTHONPATH=/home/xxx/caffe/python即可。

最后的效果如下

Reference：
http://blog.csdn.net/feynman233/article/details/71430023
http://gcucurull.github.io/caffe/python/deep-learning/2016/06/29/caffe-confusion-matrix/