CS231n Assignment2--Q2

Q2: Batch Normalization

作业代码已上传至我github: https://github.com/jingshuangliu22/cs231n,欢迎参考、讨论、指正。

BatchNormalization.ipynb

X_val: (1000, 3, 32, 32)
X_train: (49000, 3, 32, 32)
X_test: (1000, 3, 32, 32)
y_val: (1000,)
y_train: (49000,)
y_test: (1000,)

Batch normalization: Forward

Before batch normalization:
means: [ 14.07577236 -27.92959657 -33.87722636]
stds: [ 32.45504012 26.91054894 32.5339723 ]
After batch normalization (gamma=1, beta=0)
mean: [ 5.84254867e-17 2.32036612e-16 -2.39808173e-16]
std: [ 1. 0.99999999 1. ]
After batch normalization (nontrivial gamma, beta)
means: [ 11. 12. 13.]
stds: [ 1. 1.99999999 2.99999999]

After batch normalization (test-time):
means: [ 0.02347014 -0.04351107 -0.05799624]
stds: [ 1.04889033 1.01943852 1.02120248]

Batch Normalization: backward

dx error: 1.49671323478e-09
dgamma error: 3.86167816656e-11
dbeta error: 6.4892893412e-12

Batch Normalization: alternative backward

dx difference: 1.62010020328e-12
dgamma difference: 3.6199883842e-14
dbeta difference: 0.0
speedup: 0.94x

Fully Connected Nets with Batch Normalization

Running check with reg = 0
Initial loss: 2.38575331646
W1 relative error: 9.09e-06
W2 relative error: 3.44e-05
W3 relative error: 9.63e-10
b1 relative error: 1.78e-07
b2 relative error: 1.78e-07
b3 relative error: 1.23e-10
beta1 relative error: 5.59e-09
beta2 relative error: 1.50e-08
gamma1 relative error: 5.54e-09
gamma2 relative error: 7.56e-08

Running check with reg = 3.14
Initial loss: 11.8905697635
W1 relative error: 1.00e+00
W2 relative error: 1.00e+00
W3 relative error: 1.90e-07
b1 relative error: 8.88e-08
b2 relative error: 4.44e-08
b3 relative error: 3.20e-10
beta1 relative error: 3.74e-08
beta2 relative error: 6.86e-08
gamma1 relative error: 3.76e-08
gamma2 relative error: 1.54e-08

Batchnorm for deep networks

(Iteration 1 / 400) loss: 2.320615
(Epoch 0 / 20) train acc: 0.115000; val_acc: 0.112000
(Epoch 1 / 20) train acc: 0.377000; val_acc: 0.289000
(Epoch 2 / 20) train acc: 0.460000; val_acc: 0.332000
(Epoch 3 / 20) train acc: 0.542000; val_acc: 0.375000
(Epoch 4 / 20) train acc: 0.612000; val_acc: 0.349000
(Epoch 5 / 20) train acc: 0.662000; val_acc: 0.340000
(Epoch 6 / 20) train acc: 0.712000; val_acc: 0.358000
(Epoch 7 / 20) train acc: 0.747000; val_acc: 0.347000
(Epoch 8 / 20) train acc: 0.811000; val_acc: 0.361000
(Epoch 9 / 20) train acc: 0.852000; val_acc: 0.355000
(Epoch 10 / 20) train acc: 0.882000; val_acc: 0.345000
(Iteration 201 / 400) loss: 0.687450
(Epoch 11 / 20) train acc: 0.907000; val_acc: 0.359000
(Epoch 12 / 20) train acc: 0.932000; val_acc: 0.354000
(Epoch 13 / 20) train acc: 0.939000; val_acc: 0.333000
(Epoch 14 / 20) train acc: 0.953000; val_acc: 0.346000
(Epoch 15 / 20) train acc: 0.977000; val_acc: 0.342000
(Epoch 16 / 20) train acc: 0.976000; val_acc: 0.357000
(Epoch 17 / 20) train acc: 0.982000; val_acc: 0.357000
(Epoch 18 / 20) train acc: 0.986000; val_acc: 0.355000
(Epoch 19 / 20) train acc: 0.989000; val_acc: 0.341000
(Epoch 20 / 20) train acc: 0.987000; val_acc: 0.348000
(Iteration 1 / 400) loss: 2.302501
(Epoch 0 / 20) train acc: 0.120000; val_acc: 0.129000
(Epoch 1 / 20) train acc: 0.221000; val_acc: 0.205000
(Epoch 2 / 20) train acc: 0.315000; val_acc: 0.260000
(Epoch 3 / 20) train acc: 0.333000; val_acc: 0.287000
(Epoch 4 / 20) train acc: 0.352000; val_acc: 0.299000
(Epoch 5 / 20) train acc: 0.390000; val_acc: 0.305000
(Epoch 6 / 20) train acc: 0.430000; val_acc: 0.310000
(Epoch 7 / 20) train acc: 0.459000; val_acc: 0.328000
(Epoch 8 / 20) train acc: 0.495000; val_acc: 0.325000
(Epoch 9 / 20) train acc: 0.507000; val_acc: 0.328000
(Epoch 10 / 20) train acc: 0.541000; val_acc: 0.325000
(Iteration 201 / 400) loss: 1.127434
(Epoch 11 / 20) train acc: 0.607000; val_acc: 0.336000
(Epoch 12 / 20) train acc: 0.634000; val_acc: 0.327000
(Epoch 13 / 20) train acc: 0.703000; val_acc: 0.347000
(Epoch 14 / 20) train acc: 0.696000; val_acc: 0.324000
(Epoch 15 / 20) train acc: 0.764000; val_acc: 0.339000
(Epoch 16 / 20) train acc: 0.781000; val_acc: 0.315000
(Epoch 17 / 20) train acc: 0.756000; val_acc: 0.305000
(Epoch 18 / 20) train acc: 0.824000; val_acc: 0.305000
(Epoch 19 / 20) train acc: 0.842000; val_acc: 0.321000
(Epoch 20 / 20) train acc: 0.876000; val_acc: 0.315000

Batch normalization and initialization

Running weight scale 1 / 20
Running weight scale 2 / 20
Running weight scale 3 / 20
Running weight scale 4 / 20
Running weight scale 5 / 20
Running weight scale 6 / 20
Running weight scale 7 / 20
Running weight scale 8 / 20
Running weight scale 9 / 20
Running weight scale 10 / 20
Running weight scale 11 / 20
Running weight scale 12 / 20
Running weight scale 13 / 20
Running weight scale 14 / 20
Running weight scale 15 / 20
Running weight scale 16 / 20
cs231n/layers.py:773: RuntimeWarning: divide by zero encountered in log
loss = -np.sum(np.log(probs[np.arange(N), y])) / N
Running weight scale 17 / 20
Running weight scale 18 / 20
Running weight scale 19 / 20
Running weight scale 20 / 20