assignment 1

1.在需要使用bool矩阵选取元素数值时，应该将bool类型转换为float类型（此处的错误导致我debug了很久）：

a = np.array([[1,-1,3],[2,3,5],[-1,-3,4]])bool_m = a > 0b = np.sum(bool_m,axis=1)bool_m[1,:] = b #此时bool_m中的元素还是true/falseprint(bool_m)bool_m = bool_m*1.bool_m[1,:] = b #变为数值print(bool_m)

2,通过两个列表作为坐标来选取矩阵中的元素，还可以使用

a = [1,1,3]b = [2,3,1]c = np.arange(16).reshape(4,4)print(c[a,b])c[a,b] = np.arange(3)

* 两层NN的fine-tune 的训练

hyperparameter :

1. learning_rate
2. layer_size（hidden_layer_node_num）
3. traning_epoch
4. regularization

测试情况：lr = 1e-4 ,node_num=50 , traning_time = 2000*100,reg=0.25

（红色代表training acc,紫色代表testing acc(training acc仅指training batch的acc) )

存在的问题：
1. loss冷启动，表明loss没有初始化好
2. loss下降过慢，表明lr过小
3. training和testing的acc曲线交叠在一起，表明model capcity不够。

---

fine-tune过程

step1,Preprocess the data,zero center就行

step2,choose a architecture:

1,. double check the loss the reasonable(有reg和没有reg的情况)
2, try to overfitting a small portion of the training data (reg=0)
3, coarse -> fine
1) corase

iter_num = 100for it in range(iter_num):    new_net = TwoLayerNet(input_size, hidden_size, num_classes)    lr = 10**np.random.uniform(-3,-6)    reg = 10**np.random.uniform(-5,5)    stats = new_net.train(X_train, y_train, X_val, y_val,num_iters=5000, batch_size=200,            learning_rate=lr, learning_rate_decay=0.95,reg=reg, verbose=False)    val_acc = (new_net.predict(X_val) == y_val).mean()    print('lr: %e   reg: %e   Validation accuracy:%f ' %(lr,reg,val_acc))

结果如下：（因为时间原因，没有运行到100次就break了）

lr: 6.510157e-04   reg: 3.669642e+04   Validation accuracy:0.087000 lr: 2.643562e-06   reg: 1.044038e-01   Validation accuracy:0.235000 lr: 1.558228e-04   reg: 1.763366e-04   Validation accuracy:0.465000 lr: 3.269025e-05   reg: 1.024827e+00   Validation accuracy:0.309000 lr: 3.788300e-04   reg: 1.748592e-01   Validation accuracy:0.511000 lr: 8.619492e-06   reg: 2.045766e+00   Validation accuracy:0.180000 lr: 4.993109e-06   reg: 1.745723e+04   Validation accuracy:0.102000 lr: 4.883415e-04   reg: 1.855065e-01   Validation accuracy:0.521000 lr: 2.214773e-04   reg: 4.860300e-02   Validation accuracy:0.491000 lr: 2.573395e-06   reg: 3.769904e-03   Validation accuracy:0.246000 lr: 2.109817e-05   reg: 3.371226e-02   Validation accuracy:0.274000 lr: 2.708635e-04   reg: 2.103766e+03   Validation accuracy:0.098000 lr: 8.793248e-04   reg: 2.777957e+02   Validation accuracy:0.078000 lr: 7.318431e-04   reg: 4.973982e-04   Validation accuracy:0.540000 lr: 3.145155e-04   reg: 2.485930e-05   Validation accuracy:0.505000 lr: 2.975205e-04   reg: 6.164198e+00   Validation accuracy:0.451000 lr: 7.481808e-05   reg: 6.600768e+02   Validation accuracy:0.078000 lr: 9.190149e-06   reg: 1.595968e+00   Validation accuracy:0.193000 lr: 2.421630e-04   reg: 1.449934e+03   Validation accuracy:0.079000 lr: 1.483436e-06   reg: 7.021175e-03   Validation accuracy:0.205000 lr: 4.497050e-06   reg: 8.490002e+00   Validation accuracy:0.213000 lr: 1.298748e-04   reg: 5.249712e+01   Validation accuracy:0.238000 lr: 2.802625e-04   reg: 7.045640e+03   Validation accuracy:0.078000 lr: 6.116626e-06   reg: 1.250906e+01   Validation accuracy:0.205000 lr: 5.290487e-05   reg: 1.282979e-01   Validation accuracy:0.369000 lr: 4.644485e-06   reg: 2.415987e-04   Validation accuracy:0.215000 lr: 4.887054e-04   reg: 1.509442e-04   Validation accuracy:0.527000 lr: 5.136713e-05   reg: 3.751147e+01   Validation accuracy:0.258000 lr: 4.018885e-05   reg: 1.130037e-04   Validation accuracy:0.337000 lr: 2.501210e-05   reg: 1.218030e+00   Validation accuracy:0.278000 lr: 5.686478e-06   reg: 4.445485e+03   Validation accuracy:0.087000 lr: 2.776303e-04   reg: 4.061512e-01   Validation accuracy:0.491000 lr: 7.090708e-04   reg: 2.382944e+02   Validation accuracy:0.087000 lr: 1.683200e-04   reg: 9.081885e-04   Validation accuracy:0.473000 lr: 3.538319e-05   reg: 2.933638e-05   Validation accuracy:0.329000 lr: 2.639026e-05   reg: 1.571852e+02   Validation accuracy:0.087000 

找到相对比较好的结果：

lr: 4.887054e-04   reg: 1.509442e-04   Validation accuracy:0.527000lr: 7.318431e-04   reg: 4.973982e-04   Validation accuracy:0.540000 lr: 3.145155e-04   reg: 2.485930e-05   Validation accuracy:0.505000 lr: 4.883415e-04   reg: 1.855065e-01   Validation accuracy:0.521000 lr: 3.788300e-04   reg: 1.748592e-01   Validation accuracy:0.511000

**2)**fine
大致可以将lr范围确定到 e-4 ~e-3 ,reg的范围确定到 1e-4~1e-1，
sample20次的结果：

lr: 4.058940e-04   reg: 1.929913e-02   Validation accuracy:0.534000 lr: 4.222605e-04   reg: 1.200906e-02   Validation accuracy:0.503000 lr: 2.211911e-04   reg: 1.255418e-02   Validation accuracy:0.485000 lr: 2.588576e-04   reg: 8.566199e-04   Validation accuracy:0.488000 lr: 1.229843e-04   reg: 2.898743e-04   Validation accuracy:0.466000 lr: 1.985153e-04   reg: 4.509311e-04   Validation accuracy:0.473000 lr: 1.175998e-04   reg: 1.462556e-03   Validation accuracy:0.453000 lr: 6.840469e-04   reg: 2.621410e-04   Validation accuracy:0.520000 lr: 5.093372e-04   reg: 1.555551e-03   Validation accuracy:0.525000 lr: 2.212163e-04   reg: 2.355791e-03   Validation accuracy:0.495000 lr: 2.239080e-04   reg: 2.024601e-02   Validation accuracy:0.495000 lr: 2.805192e-04   reg: 4.729087e-02   Validation accuracy:0.519000 lr: 2.400963e-04   reg: 2.614076e-03   Validation accuracy:0.484000 lr: 2.626168e-04   reg: 3.921650e-02   Validation accuracy:0.498000 lr: 4.159985e-04   reg: 1.809018e-04   Validation accuracy:0.515000 lr: 2.357435e-04   reg: 6.627438e-03   Validation accuracy:0.496000 lr: 6.429720e-04   reg: 5.428195e-04   Validation accuracy:0.507000 lr: 1.941110e-04   reg: 5.121491e-04   Validation accuracy:0.470000 lr: 2.136214e-04   reg: 1.695283e-03   Validation accuracy:0.482000 lr: 4.427498e-04   reg: 9.225977e-03   Validation accuracy:0.527000 

以及上一次的最好结果

lr: 7.318431e-04   reg: 4.973982e-04   Validation accuracy:0.540000 

相比上一次sample，这次明显更好，比较好的选择(0,7,8,19)从左到右，从上到下的四副图像如下：

最后选择第19个组合，因为上升空间较大，还没有趋于平缓。

最终结果：

lr: 4.427498e-04   reg: 9.225977e-03   Validation accuracy:0.527000 

batch=200,node_num=100 训练15000次后，在validation上的结果为52.1%,在test上测试,acc为52.9%。但是由图可知，存在overfitting现象所以我们再尝试训练7500次，10000次左右。结果如下：

val acc = 0.541,test acc = 0.537

val acc = 0.54,test acc = 0.543

综上选择取得最好的validation acc的model :batch=200,node_num=100,training_time=7500

改进方向：

1. W的初始化采用Xavier
2. 使用更深的网络结构

Note:Step2中try to overfitting a small portion of the training data (reg=0)

使用Xavier的情况

使用std*randn的情况
但是奇怪的是在后续过程中，我不小心使用了std*randn的方式，也得到了很好的结果，而且我又重新复现了一次，究其原因我也不是太清楚。这会不会表明使用Xavier会得到更好的结果呢？
于是决定使用Xavier初始化方法再重新做一次实验

lr: 6.341384e-05   reg: 1.769403e-03   Validation accuracy:0.335000 0lr: 3.852774e-04   reg: 3.193370e-04   Validation accuracy:0.384000 lr: 2.094796e-06   reg: 5.271914e-05   Validation accuracy:0.260000 lr: 3.409431e-04   reg: 6.048618e-02   Validation accuracy:0.378000 lr: 2.334389e-06   reg: 1.421406e-04   Validation accuracy:0.245000 lr: 1.074157e-06   reg: 4.755761e-01   Validation accuracy:0.222000 lr: 4.507436e-05   reg: 3.534975e-03   Validation accuracy:0.304000 lr: 5.701219e-05   reg: 2.793682e+01   Validation accuracy:0.337000 lr: 1.424817e-04   reg: 1.072272e-05   Validation accuracy:0.337000 lr: 3.476407e-06   reg: 2.268886e+01   Validation accuracy:0.257000 lr: 3.107524e-05   reg: 3.554090e-03   Validation accuracy:0.302000 lr: 7.066331e-05   reg: 8.943052e-04   Validation accuracy:0.294000 lr: 7.147195e-06   reg: 6.824755e+01   Validation accuracy:0.335000 lr: 2.840902e-06   reg: 4.868638e-02   Validation accuracy:0.272000 lr: 3.801703e-05   reg: 2.096270e+01   Validation accuracy:0.442000  14lr: 4.093321e-06   reg: 2.500971e-01   Validation accuracy:0.269000 lr: 2.001587e-05   reg: 5.745243e+01   Validation accuracy:0.349000 lr: 6.730932e-04   reg: 1.897193e+00   Validation accuracy:0.509000  17lr: 6.049389e-04   reg: 4.889209e-04   Validation accuracy:0.455000 lr: 1.512126e-05   reg: 1.814202e+00   Validation accuracy:0.320000 lr: 3.772819e-05   reg: 2.384045e-01   Validation accuracy:0.313000 lr: 8.231728e-04   reg: 5.110095e+00   Validation accuracy:0.462000 lr: 6.948852e-05   reg: 1.765428e-02   Validation accuracy:0.367000 lr: 5.072889e-05   reg: 1.018637e-01   Validation accuracy:0.355000 lr: 2.877371e-05   reg: 7.400674e-02   Validation accuracy:0.312000 lr: 2.735792e-04   reg: 4.399957e-04   Validation accuracy:0.371000 lr: 2.983375e-05   reg: 3.060068e-01   Validation accuracy:0.315000 lr: 5.892237e-05   reg: 2.432885e+00   Validation accuracy:0.335000 lr: 1.810665e-04   reg: 3.647910e-01   Validation accuracy:0.381000 lr: 1.017972e-05   reg: 4.084320e-01   Validation accuracy:0.308000 30th  lr: 4.322279e-06   reg: 2.016450e-03   Validation accuracy:0.267000 31th  lr: 1.052033e-04   reg: 8.066398e-02   Validation accuracy:0.329000 32th  lr: 1.940260e-04   reg: 3.866905e-01   Validation accuracy:0.380000 33th  lr: 2.908686e-04   reg: 2.485808e+01   Validation accuracy:0.334000 34th  lr: 6.938006e-04   reg: 3.538238e-02   Validation accuracy:0.431000 35th  lr: 1.897502e-04   reg: 2.381476e-04   Validation accuracy:0.354000 36th  lr: 1.028106e-04   reg: 2.249541e-01   Validation accuracy:0.365000 37th  lr: 9.773299e-06   reg: 4.400711e-04   Validation accuracy:0.319000 38th  lr: 8.901756e-05   reg: 6.154393e+01   Validation accuracy:0.237000 39th  lr: 2.382143e-04   reg: 3.235230e-02   Validation accuracy:0.376000 40th  lr: 6.426793e-04   reg: 2.050400e-01   Validation accuracy:0.480000 41th  lr: 3.168815e-04   reg: 5.118725e+00   Validation accuracy:0.463000 42th  lr: 6.664813e-04   reg: 4.848741e-01   Validation accuracy:0.469000 43th  lr: 8.270327e-04   reg: 2.412895e-03   Validation accuracy:0.431000 44th  lr: 3.473371e-05   reg: 1.298962e-02   Validation accuracy:0.321000 45th  lr: 5.198179e-05   reg: 3.302815e+01   Validation accuracy:0.321000 46th  lr: 2.981851e-05   reg: 4.685633e-05   Validation accuracy:0.327000 47th  lr: 1.331836e-06   reg: 1.185023e-03   Validation accuracy:0.225000 48th  lr: 2.388945e-05   reg: 1.057670e+01   Validation accuracy:0.337000 49th  lr: 1.171363e-04   reg: 1.908012e-01   Validation accuracy:0.354000 50th  lr: 5.786395e-06   reg: 4.114150e-05   Validation accuracy:0.270000 51th  lr: 4.908611e-04   reg: 1.332642e-04   Validation accuracy:0.440000 52th  lr: 1.825870e-06   reg: 1.507947e-01   Validation accuracy:0.248000 53th  lr: 8.423663e-06   reg: 1.810298e-05   Validation accuracy:0.304000 54th  lr: 3.539308e-06   reg: 1.211471e-05   Validation accuracy:0.271000 55th  lr: 1.768898e-06   reg: 1.316256e-02   Validation accuracy:0.238000 56th  lr: 1.910679e-05   reg: 4.767425e-03   Validation accuracy:0.328000 57th  lr: 2.361348e-04   reg: 1.944416e-05   Validation accuracy:0.383000 58th  lr: 6.626397e-05   reg: 5.926622e+01   Validation accuracy:0.244000 59th  lr: 7.005012e-04   reg: 5.000929e-03   Validation accuracy:0.434000 

相对比较好的点

lr: 3.801703e-05   reg: 2.096270e+01   Validation accuracy:0.442000 lr: 6.730932e-04   reg: 1.897193e+00   Validation accuracy:0.509000 ok 17lr: 6.049389e-04   reg: 4.889209e-04   Validation accuracy:0.455000 ok 18lr: 8.231728e-04   reg: 5.110095e+00   Validation accuracy:0.462000 40th  lr: 6.426793e-04   reg: 2.050400e-01   Validation accuracy:0.480000 ok41th  lr: 3.168815e-04   reg: 5.118725e+00   Validation accuracy:0.463000 42th  lr: 6.664813e-04   reg: 4.848741e-01   Validation accuracy:0.469000 ok51th  lr: 4.908611e-04   reg: 1.332642e-04   Validation accuracy:0.440000 ok

lr=1e-4~1e-3,1e-4~1e-360th  lr: 4.709525e-04   reg: 8.095851e-04   Validation accuracy:0.439000 61th  lr: 2.726294e-04   reg: 1.949236e-04   Validation accuracy:0.377000 62th  lr: 4.297156e-04   reg: 4.621484e-04   Validation accuracy:0.443000 63th  lr: 2.489500e-04   reg: 6.292587e-04   Validation accuracy:0.375000 64th  lr: 2.331295e-04   reg: 2.676282e-04   Validation accuracy:0.369000 65th  lr: 2.781939e-04   reg: 1.396553e-04   Validation accuracy:0.365000 66th  lr: 9.496852e-04   reg: 4.296325e-04   Validation accuracy:0.438000 67th  lr: 7.280289e-04   reg: 4.663365e-04   Validation accuracy:0.459000 68th  lr: 5.814480e-04   reg: 2.177770e-04   Validation accuracy:0.457000 69th  lr: 5.976867e-04   reg: 3.075948e-04   Validation accuracy:0.455000 70th  lr: 1.489146e-04   reg: 4.210730e-04   Validation accuracy:0.337000 71th  lr: 5.585008e-04   reg: 6.217684e-04   Validation accuracy:0.437000 72th  lr: 4.433013e-04   reg: 1.457733e-04   Validation accuracy:0.450000 73th  lr: 2.387401e-04   reg: 8.833313e-04   Validation accuracy:0.371000 74th  lr: 2.781313e-04   reg: 6.748215e-04   Validation accuracy:0.396000 75th  lr: 1.204839e-04   reg: 2.052842e-04   Validation accuracy:0.344000 76th  lr: 8.505226e-04   reg: 2.182052e-04   Validation accuracy:0.446000 77th  lr: 3.224040e-04   reg: 1.535878e-04   Validation accuracy:0.406000 78th  lr: 5.363216e-04   reg: 1.629328e-04   Validation accuracy:0.437000 79th  lr: 3.087325e-04   reg: 7.169442e-04   Validation accuracy:0.420000 80th  lr: 1.664995e-04   reg: 9.278335e-04   Validation accuracy:0.392000 81th  lr: 6.931702e-04   reg: 1.442553e-04   Validation accuracy:0.443000 82th  lr: 3.088621e-04   reg: 2.243105e-04   Validation accuracy:0.384000 83th  lr: 4.278311e-04   reg: 1.159277e-04   Validation accuracy:0.409000 84th  lr: 1.001049e-04   reg: 4.231650e-04   Validation accuracy:0.321000 85th  lr: 3.351125e-04   reg: 8.224279e-04   Validation accuracy:0.410000 86th  lr: 9.641541e-04   reg: 9.183995e-04   Validation accuracy:0.465000 87th  lr: 1.260288e-04   reg: 4.936542e-04   Validation accuracy:0.331000 88th  lr: 7.027471e-04   reg: 4.970635e-04   Validation accuracy:0.459000 89th  lr: 1.710624e-04   reg: 1.329212e-04   Validation accuracy:0.341000 lr=1e-4~1e-3,reg=1e-1~1e190th  lr: 1.887898e-04   reg: 5.492338e-01   Validation accuracy:0.369000 91th  lr: 6.789602e-04   reg: 2.696616e-01   Validation accuracy:0.485000 92th  lr: 9.224811e-04   reg: 8.581825e-01   Validation accuracy:0.522000 93th  lr: 1.916202e-04   reg: 2.981688e+00   Validation accuracy:0.502000 94th  lr: 4.389360e-04   reg: 2.801716e+00   Validation accuracy:0.491000 95th  lr: 5.546313e-04   reg: 4.809618e-01   Validation accuracy:0.485000 96th  lr: 2.543303e-04   reg: 1.886383e+00   Validation accuracy:0.485000 97th  lr: 5.447017e-04   reg: 5.978606e-01   Validation accuracy:0.485000 98th  lr: 7.483245e-04   reg: 8.129600e+00   Validation accuracy:0.439000 99th  lr: 1.272105e-04   reg: 2.349757e+00   Validation accuracy:0.404000 100th  lr: 5.265308e-04   reg: 2.310304e-01   Validation accuracy:0.468000 101th  lr: 2.381013e-04   reg: 1.190625e-01   Validation accuracy:0.392000 102th  lr: 3.632505e-04   reg: 1.754443e+00   Validation accuracy:0.493000 103th  lr: 6.462288e-04   reg: 7.055405e-01   Validation accuracy:0.496000 104th  lr: 6.799179e-04   reg: 1.057823e-01   Validation accuracy:0.455000 105th  lr: 1.349350e-04   reg: 1.536159e+00   Validation accuracy:0.377000 106th  lr: 5.687849e-04   reg: 1.006980e+00   Validation accuracy:0.506000 107th  lr: 2.301599e-04   reg: 5.041087e+00   Validation accuracy:0.457000 108th  lr: 1.208513e-04   reg: 4.028463e-01   Validation accuracy:0.316000 109th  lr: 1.854946e-04   reg: 5.846844e+00   Validation accuracy:0.443000 110th  lr: 3.002745e-04   reg: 2.760922e+00   Validation accuracy:0.495000 111th  lr: 2.479276e-04   reg: 6.671733e-01   Validation accuracy:0.415000 112th  lr: 1.645383e-04   reg: 3.215961e-01   Validation accuracy:0.347000 113th  lr: 2.039684e-04   reg: 1.474140e-01   Validation accuracy:0.360000 114th  lr: 1.500013e-04   reg: 2.389144e-01   Validation accuracy:0.333000 115th  lr: 3.748992e-04   reg: 3.976034e-01   Validation accuracy:0.440000 116th  lr: 6.600451e-04   reg: 2.083297e-01   Validation accuracy:0.488000 117th  lr: 5.128263e-04   reg: 8.254960e+00   Validation accuracy:0.431000 118th  lr: 1.038475e-04   reg: 2.729728e-01   Validation accuracy:0.305000 119th  lr: 3.601892e-04   reg: 4.204359e-01   Validation accuracy:0.422000 

选择

92th  lr: 9.224811e-04   reg: 8.581825e-01   Validation accuracy:0.522000  (batch_size=200)

训练20,000次后，

Validation accuracy:0.554000 Test accuracy:  0.551 

总结：

*做实验时及时记下代码的改变，在sublime中改变代码时要保存