Stanford ML - Lecture 5 - Neural Networks: Learning

来源：互联网发布：芭芭拉史翠珊知乎编辑：程序博客网时间：2024/04/28 00:55

1. Cost function

Neural Network (Classification)

$\{(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), \cdots, (x^{(m)}, y^{(m)})\}$
$L = \textrm{total no. of layers in network}$
$s_l = \textrm{no. of units (not counting bias unit) in layer} \ l$

Binary classification
- $y = 0 \ \textrm{or} \ 1$
- 1 output unit
Multi-class classification (K classes)
- $y \in \mathbb{R}^K$
- K output units

Cost function
- Logistic regression:

$J(\theta) = - \frac{1}{m} \left[ \sum_{i=1}^m y^{(i)} \log h_{\theta}(x^{(i)}) + (1 - y^{(i)}) \log (1 - h_{\theta}(x^{(i)})) \right ] + \frac{\lambda}{2m} \sum_{j=1}^n \theta_j^2$

Neural network:

$h_{\theta}(x) \in \mathbb{R}^K \qquad (h_{\theta}(x))_i = i^{th} \ \textrm{output}$

$\begin{array}{ll} J(\theta) = - \frac{1}{m} \left[ \sum_{i=1}^m \sum_{k=1}^K y_k^{(i)} \log (h_{\theta}(x^{(i)}))_k + (1 - y_k^{(i)}) \log (1 - (h_{\theta}(x^{(i)}))_k) \right ] + \frac{\lambda}{2m} \sum_{l=1}^{L-1} \sum_{i=1}^{s_l} \sum_{j=1}^{s_{l+1}} (\theta_{ji}^{(l)})^2 \end{array}$

2. Backpropagation algorithm

Gradient descent
- need code to compute
  - $J(\theta)$
  - $\frac{\partial }{\partial \theta_{ij}^{(l)}} J(\theta)$

Is it $\frac{\partial }{\partial \theta_{ji}^{(l)}} J(\theta)$ ??

Gradient computation

然后根据backpropagation算法进行梯度的计算，这里引入了error变量δ，用来表示真实值与forward propagation计算值之间的差，也是梯度的主要依据来源。

我们定义神经网络的总误差为：

$E = \frac{1}{2}\sum_{i}{(y_i-a_i)^2}$ 希望通过调整权重参数W（也就是theta）来最小化E。假设一个共有3层的神经网络，在最后一层，由于

$\Delta W \propto -\frac{\partial E}{\partial W}$ $\left( \Delta \theta_k = -\xi \frac{\partial E}{\partial \theta_k} \right )$

而对于前面的一层（如第三层），其误差可以定义为：

$\Delta \Theta_{k-1} = \frac{\partial E}{\partial a_k}\cdot \frac{\partial a_k}{\partial z_k} \cdot \frac{\partial z_k}{\Theta _{k-1}}$

分别代入即得

$\frac{\partial E}{\partial a_k} = a_k-y \\ \frac{\partial a_k}{\partial z_k} = \frac{\partial g(z_k))}{\partial z_k} = \frac{e^{-z}}{(1+e^{-z})^2} = a_k(1-a_k)\\ \frac{\partial z_k}{\partial \Theta _{k-1}} = a_{k-1}$

由此得来\theta_{k}的update方程： $\Theta_{k} = \xi (y-a_k)a_k(1-a_k)a_{k-1}$

如果将误差对激励函数（activation function）的导数记做δ，则有：

$\delta_{k} = (y-a_k)a_k(1-a_k)$

$\Delta\Theta_k = \xi \delta_k \cdot a_{k-1}$

对于前面一层 ,更新同理， $\Delta\Theta_k = \xi \delta_k \cdot a_{k-1}$ ，只是上一层\Theta梯度的第一个分量E对a_k求导有所变化，

$\begin{align*} \frac{\partial E}{\partial a_{j}}=\sum_{k} \frac{\partial E}{\partial a_k}\cdot \frac{\partial a_k}{\partial z_k}\cdot \frac{\partial z_k}{\partial a_{j}}\\ = \sum_{k}(y-a_k)\cdot a_k(1-a_k)\cdot \Theta_j \end{align*}$

但是 $\Delta\Theta_k = \xi \delta_k \cdot a_{k-1}$ 始终是不变的。

From: http://blog.csdn.net/abcjennifer/article/details/7758797