Deep Learning:深度前馈神经网络(五)
来源:互联网 发布:mac下安装配置redis4 编辑:程序博客网 时间:2024/05/17 08:21
Back-Propagation and Other Differentiation Algorithms
- Forward propagation: The inputs x provide the initial information that then propagates up to the hidden units at each layer and finally produces
y^ . - Back-propagation: allows the information from the cost to then flow backwards through the network, in order to compute the gradient.
- The term back-propagation is often misunderstood as meaning the whole learning algorithm for multi-layer neural networks. Actually, back-propagation refers only to the method for computing the gradient, while another algorithm, such as stochastic gradient descent, is used to perform learning using this gradient.
- In learning algorithms, the gradient we most often require is the gradient of the cost function with respect to the parameters,
∇θJ(θ) .
Computational Graphs
To describe the back-propagation algorithm more precisely, it is helpful to have a more precise computational graph language.
- Here, we use each node in the graph to indicate a variable. The variable may be a scalar, vector, matrix, tensor, or even a variable of another type.
- An operation is a simple function of one or more variables. Our graph language is accompanied by a set of allowable operations. Functions more complicated than the operations in this set may be described by composing many operations together.
Chain Rule of Calculus
The chain rule of calculus (not to be confused with the chain rule of probability) is used to compute the derivatives of functions formed by composing other functions whose derivatives are known. Back-propagation is an algorithm that computes the chain rule, with a specific order of operations that is highly efficient.
Let x be a real number, and let f and g both be functions mapping from a real number to a real number. Suppose that
We can generalize this beyond the scalar case:
In vector notation, this may be equivalently written as
where
From this we see that the gradient of a variable x can be obtained by multiplying a Jacobian matrix
阅读全文
0 0
- Deep Learning:深度前馈神经网络(五)
- Deep Learning:深度前馈神经网络(一)
- Deep Learning:深度前馈神经网络(二)
- Deep Learning : 深度前馈神经网络(三)
- Deep Learning:深度前馈神经网络(四)
- Deep Learning深度学习之(五)CNN卷积神经网络
- 深度学习(Deep Learning)读书思考二:前向神经网络
- 深度学习(Deep Learning)读书思考五:卷积神经网络(CNN)
- 【五】Deep Learning模型之:CNN卷积神经网络(一)深度解析CNN
- Deep Learning(深度学习)Convolutional Neural Networks卷积神经网络
- Deep Learning(深度学习)Convolutional Neural Networks卷积神经网络
- 深度学习Deep Learning(01)_CNN卷积神经网络
- 深度神经网络优化(一)- Practical aspects of Deep Learning
- Deep Learning(深度学习)之(五)神经网络训练中的高效BP(反向传播算法)
- 深度学习方法(五):卷积神经网络CNN经典模型整理Lenet,Alexnet,Googlenet,VGG,Deep Residual Learning
- 深度学习方法(五):卷积神经网络CNN经典模型整理Lenet,Alexnet,Googlenet,VGG,Deep Residual Learning
- 深度学习方法(五):卷积神经网络CNN经典模型整理Lenet,Alexnet,Googlenet,VGG,Deep Residual Learning
- 深度学习方法(五):卷积神经网络CNN经典模型整理Lenet,Alexnet,Googlenet,VGG,Deep Residual Learning
- javascript判断是否是QQ或微信手机内置浏览器
- iOS/Android 微信及浏览器中唤起本地APP
- SpringMVC工作原理
- 深入理解Ribbon之源码解析
- zookeeper学习记录-06 Zookeeper JavaAPI操作
- Deep Learning:深度前馈神经网络(五)
- 一周五次课(10月20日)
- tornado学习之路
- 自定义View笔记(三) --- ViewDragHelper
- 演示文件IO的例子
- Linux终端部分快捷方式
- C++ 遗传学SFLA混合蛙跳算法
- PopupWindow在7.0上显示的坑
- linux 下 启动mysql服务出错!Starting MySQL.The server quit without updating PID file (XXX.pid)