《neural network and deep learning》题解——ch03 其他技术(momentun,tanh)
来源:互联网 发布:瑞虎3轮毂数据 编辑:程序博客网 时间:2024/05/22 13:05
http://blog.csdn.net/u011239443/article/details/77848503
问题一
• 如果我们使⽤ µ > 1 会有什么问题?
• 如果我们使⽤ µ < 0 会有什么问题?
如果我们使⽤ µ > 1,∇C趋近于0时,v依旧会越来越大。如果我们使⽤ µ < 0,∇C趋近于0时,v会变为梯度的反方向。
问题二
增加基于 momentum 的随机梯度下降到 network2.py 中。
增加 b_velocity 和 w_velocity
def default_weight_initializer(self): self.biases = [np.random.randn(y, 1) for y in self.sizes[1:]] self.weights = [np.random.randn(y, x)/np.sqrt(x) for x, y in zip(self.sizes[:-1], self.sizes[1:])] self.b_velocity = [np.random.randn(y, 1) for y in self.sizes[1:]] self.w_velocity = [np.random.randn(y, x)/np.sqrt(x) for x, y in zip(self.sizes[:-1], self.sizes[1:])]
增加参数 µ (mu)
def SGD(self, training_data, epochs, mini_batch_size, eta, lmbda = 0.0,mu = 1.0, evaluation_data=None, monitor_evaluation_cost=False, monitor_evaluation_accuracy=False, monitor_training_cost=False, monitor_training_accuracy=False): ...... for j in xrange(epochs): random.shuffle(training_data) mini_batches = [ training_data[k:k+mini_batch_size] for k in xrange(0, n, mini_batch_size)] for mini_batch in mini_batches: self.update_mini_batch( mini_batch, eta, lmbda,mu, len(training_data)) ......
def update_mini_batch(self, mini_batch, eta, lmbda,mu, n): nabla_b = [np.zeros(b.shape) for b in self.biases] nabla_w = [np.zeros(w.shape) for w in self.weights] for x, y in mini_batch: delta_nabla_b, delta_nabla_w = self.backprop(x, y) nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)] nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)] self.w_velocity = [mu*v-(eta/len(mini_batch))*nw for v, nw in zip(self.w_velocity, nabla_w)] self.weights = [(1-eta*(lmbda/n))*w + v for w, v in zip(self.weights, self.w_velocity)] self.b_velocity = [mu*v-(eta/len(mini_batch))*nb for v, nb in zip(self.b_velocity, nabla_b)] self.biases = [b + v for b, v in zip(self.biases, self.b_velocity)]
问题三
证明公式 (111)
阅读全文
0 0
- 《neural network and deep learning》题解——ch03 其他技术(momentun,tanh)
- 《neural network and deep learning》题解——ch03 交叉熵代价函数
- 《neural network and deep learning》题解——ch03 过度拟合&规范化&权重初始化
- 《neural network and deep learning》题解——ch03 如何选择神经网络的超参数
- 《neural network and deep learning》题解——ch03 再看手写识别问题题解与源码分析
- 《neural network and deep learning》题解——ch01 神经网络
- 《neural network and deep learning》题解——ch02 Network源码分析
- 《neural network and deep learning》题解——ch02 反向传播
- Neural Network and Deep Learning
- Deep Learning——CNN(Convolution Neural Network)
- neural network and deep learning笔记(1)
- neural network and deep learning笔记(2)
- 《Neural network and deep learning》学习笔记(一)
- neural network and deep learning (1)
- neural network and deep learning (2)
- neural network and deep learning(笔记二)
- neural-networks-and-deep-learning network.py
- Neural Network and deep learning(二)
- Realm for Android 使用入门
- 四个数字:1、2、3、4,能组成多少个互不
- jvm内存溢出实例一
- Dmitry
- 《Java核心技术(卷一)》读书笔记——第六章:内部类
- 《neural network and deep learning》题解——ch03 其他技术(momentun,tanh)
- __str__
- HDU 6085 Rikka with Candies(bitset)
- 插入排序-算法导论学习
- Spring security的图片验证码方案一
- python 与机器学习实战(何宇健)代码下载
- stm32 内存问题
- ubuntu下jdk1.8的配置
- 迅为iTOP-4412开发板开机测试、电源、拨码开关、进不去uboot模式问题