机器学习笔记——第3篇

文章作者是我的一位大神学长陈雨学长~，征得学长同意后将文章转载到了我的blog上，特在此感谢学长~

代码在学长的github上：
https://github.com/unnamed2/MDL
欢迎猛击

自己实现的机器学习(3):

3.0 开始写代码

3.0.1 代码不多,去掉矩阵运算和数据读取之外只有不到150行,在这里全部拿上来:

template<size_t Row,size_t Col>void misnt_data_reader(std::vector<std::pair<Matrix, Matrix>>& outputs){/*...*/}//读取Minst数据集,读到outputs里面,对于每一张图,将其读成Row * Col的矩阵,存到pair的first 和该数据的标签值(这个图片上面是0-9中每一个数字的概率,非1即0) 详细代码右转github//MINST数据集中有60000张手写数字的图片,每张都是28 * 28的灰度图 我们把前50000张作为训练集,训练模型用,后10000张用来进行测试评估,看看现在学的咋样了//0-0.001之间的随机数,用这种初始化方法很不好,之后会有更好的初始化parameters的方法float Random_0_1() {    return (float)rand() / (float)RAND_MAX * 0.001f;}//激活函数sigmoid和sigmoid的导函数float Sigmoid(float x){    return 1.0f / (expf(-x) + 1.0f);}float Sigmoid_P(float x){    return Sigmoid(x)*(1.0f - Sigmoid(x));}//激活函数LeaklyReLU和他的导函数float LeaklyReLU(float x) {    if (x > 0.0f)return x;    return 0.1f * x;}float LeaklyReLU_P(float x) {    if (x > 0)return 1.0f;    return 0.1f;}//开始啦int main(){  //读取数据    std::vector<std::pair<Matrix, Matrix>> datas;    printf("Initializing ...");    misnt_data_reader<28 * 28, 1>(datas);    printf("done.\ttraining...\n");  //训练参数:  //我们每计算25张图更新一次参数    const int sample_per_minibatch = 25;  //中间的输出的向量的维度    const int hiddens = 100;  //学习率 这里是0.01    const float learning_rate = 0.01f / sample_per_minibatch;  //训练的参数啊 parameters 用随机数初始化    Matrix W0(hiddens, 28 * 28, Random_0_1);    Matrix B0(hiddens, 1, Random_0_1);    Matrix W1(10, hiddens, Random_0_1);    Matrix B1(10,1, Random_0_1);    std::random_device rd;    std::mt19937 g(rd());  //保存偏导数的和的矩阵 全部初始化为0    Matrix partlW0(hiddens, 28 * 28, []() {return 0.0f; });    Matrix partlB0(hiddens, 1, []() {return 0.0f; });    Matrix partlW1(10, hiddens, []() {return 0.0f; });    Matrix partlB1(10, 1, []() {return 0.0f; });    //training开始训练    //每张图片训练20次    for (int i = 0; i < 20; i++) {        auto tp = std::chrono::system_clock::now();//当前时间        std::random_shuffle(datas.begin(), datas.begin() + 50000);//先把训练集随机打乱        int DataOffset = 0;        int ErrC = 0;        for (int j = 0; j < 50000 / sample_per_minibatch; j++) {            //清空偏导数的数组 全部变成0            partlB0.SetValue(0.0f);            partlB1.SetValue(0.0f);            partlW0.SetValue(0.0f);            partlW1.SetValue(0.0f);            //对于每一张图片            for (int k = 0; k < sample_per_minibatch; k++) {                auto & samp = datas[DataOffset + k];                //向前传播的过程,即计算输出和误差                Matrix A0 = W0 * samp.first + B0;                Matrix Z0 = A0.Apply(Sigmoid);//对A0的每一个元素计算Sigmoid 懒得写重载函数                Matrix A1 = W1 * Z0 + B1;                Matrix Y = A1.Apply(LeaklyReLU);                int idx = 0;//看看我们输出的10个数字中那个数字的值最大 那样我们就认为这个数字是最有可能的                for (int i = 0; i < 10; i++) {                    if (Y(i, 0) > Y(idx, 0))idx = i;                }                if (samp.second(idx, 0) < 0.9f)ErrC++;//然而我们的结果和给定的标签并不一致的话,统计一下跪了几个                //backward pass反向传播的过程                Matrix Loss = Y-samp.second;//err 对 Y的 偏导数的值              //ElementMultiplyWith是逐元素求乘法                Loss.ElementMultiplyWith(A1.Apply(LeaklyReLU_P));//err 对 A1 的偏导数的值                partlB1 += Loss;//对B1的偏导数等于对A1的偏导数                partlW1 += Loss * ~Z0;//对W1的偏导数等于对A1的偏导数乘上Z0的逆矩阵                Loss = (~W1 * Loss); //求对Z0的偏导数                Loss.ElementMultiplyWith(A0.Apply(Sigmoid_P));//求对A0的偏导数                partlB0 += Loss;//求对B0的偏导数                partlW0 += Loss * ~samp.first;//求对W0的偏导数            }            // 更新            W0 -= partlW0 * learning_rate;            B0 -= partlB0 * learning_rate;            W1 -= partlW1 * learning_rate;            B1 -= partlB1 * learning_rate;        }        DataOffset += sample_per_minibatch;        auto ed = std::chrono::system_clock::now();//统计时间        auto g = ed - tp;        //用测试集测一下准确率        int errCount = 0;        for (int j = 0; j < 10000; j++) {            auto & samp = datas[50000 + j];            //暴力计算就可以了 求输出Y            Matrix A0 = W0 * samp.first + B0;            Matrix Z0 = A0.Apply(Sigmoid);            Matrix A1 = W1 * Z0 + B1;            Matrix Y = A1.Apply(LeaklyReLU);            int idx = 0;            for (int i = 0; i < 10; i++) {                if (Y(i, 0) > Y(idx, 0))idx = i;            }            if (samp.second(idx, 0) < 0.9f)errCount++;//GG了要统计        }        printf("Training %d / 20,loss %f%% on training set,loss %f%% on trst set,training cost %d ms\n", i+1,ErrC / 500.0f, errCount / 100.0f,std::chrono::duration_cast<std::chrono::duration<int,std::milli>>(g).count());    }}

3.0.2 运行结果

Training 1 / 20,loss 75.921997% on training set,loss 87.839996% on trst set,training cost 5986 msTraining 2 / 20,loss 58.952000% on training set,loss 61.270000% on trst set,training cost 5982 msTraining 3 / 20,loss 30.878000% on training set,loss 50.520000% on trst set,training cost 5961 msTraining 4 / 20,loss 12.102000% on training set,loss 38.549999% on trst set,training cost 5979 msTraining 5 / 20,loss 4.478000% on training set,loss 43.619999% on trst set,training cost 5801 msTraining 6 / 20,loss 6.040000% on training set,loss 37.169998% on trst set,training cost 5911 msTraining 7 / 20,loss 9.270000% on training set,loss 33.570000% on trst set,training cost 5886 msTraining 8 / 20,loss 4.076000% on training set,loss 32.689999% on trst set,training cost 5920 msTraining 9 / 20,loss 5.840000% on training set,loss 28.670000% on trst set,training cost 6021 msTraining 10 / 20,loss 9.454000% on training set,loss 29.030001% on trst set,training cost 6032 msTraining 11 / 20,loss 1.994000% on training set,loss 27.410000% on trst set,training cost 5956 msTraining 12 / 20,loss 7.128000% on training set,loss 28.510000% on trst set,training cost 5881 msTraining 13 / 20,loss 6.376000% on training set,loss 25.799999% on trst set,training cost 5828 msTraining 14 / 20,loss 4.098000% on training set,loss 28.410000% on trst set,training cost 5866 msTraining 15 / 20,loss 8.378000% on training set,loss 28.200001% on trst set,training cost 5681 msTraining 16 / 20,loss 3.168000% on training set,loss 25.540001% on trst set,training cost 5814 msTraining 17 / 20,loss 4.834000% on training set,loss 26.549999% on trst set,training cost 5744 msTraining 18 / 20,loss 0.086000% on training set,loss 24.250000% on trst set,training cost 5901 msTraining 19 / 20,loss 8.382000% on training set,loss 25.600000% on trst set,training cost 5863 msTraining 20 / 20,loss 4.632000% on training set,loss 31.549999% on trst set,training cost 5776 ms

3.1 过拟合

从上面的结果里面可以明显的看到,在训练集上的误差要比测试集小得多,甚至第18次训练训练集上准确率已经高达99.9%以上,但是在测试集上依然只有75%左右的正确率. 换句话说,神经网络已经开始玩”背答案”,对教过她的其他内容掌握的明显不好.

在机器学习的方法里面,有很多手段来避免过拟合,后面会告诉大家如何真真正正的训练出一个不管在哪正确率都在99.9%以上的算法.