cs231n assignment(1.4):two_layer_net

two_layer_net

这个练习中，使用一个两层全连接的神经网络用以分类工作
各层的情况是：
input - fully connected layer - ReLU - fully connected layer - softmax

两层的神经网络计算loss和梯度

def ReLU(x):        #ReLU non-linearity.       return np.maximum(0, x)

  def loss(self, X, y=None, reg=0.0):    # Unpack variables from the params dictionary    W1, b1 = self.params['W1'], self.params['b1']    W2, b2 = self.params['W2'], self.params['b2']    N, D = X.shape    # Compute the forward pass    scores = None    s1 = X.dot(W1)+b1    h1 = ReLU(s1)    scores = h1.dot(W2)+b2        # If the targets are not given then jump out, we're done    if y is None:      return scores    # Compute the loss    loss = None    f_max = np.max(scores,axis=1,keepdims=True)    f_scores = scores - f_max    prob = np.exp(f_scores)/np.sum(np.exp(f_scores),axis=1,keepdims=True)    loss = np.sum(-np.log(prob[np.arange(N),y]))    loss = loss/N + 0.5*reg*np.sum(W1*W1) + 0.5*reg*np.sum(W2*W2)    # Backward pass: compute gradients    grads = {}    dscores = prob    dscores[np.arange(N),y] -= 1    dscores /= N    #Second Layer    dh1 = dscores.dot(W2.T)    dW2 = h1.T.dot(dscores)    db2 = np.sum(dscores,axis=0)    #ReLU    dh1[s1<=0]=0    ds1 = dh1    #First Layer    dW1 = np.dot(X.T,ds1)    db1 = np.sum(ds1,axis=0)    #Reg    dW1 += reg*W1    dW2 += reg*W2    #Store    grads['W1'] = dW1    grads['W2'] = dW2    grads['b1'] = db1    grads['b2'] = db2                 return loss, grads

关于调试超参数：
loss的下降类似于线性，意味着学习率可能太低。训练和验证集精确度之间没有gap,说明模型capacity低，所以我们应该增大模型的size.因为更大的模型会更overfitting。