backpropagation
来源:互联网 发布:禁止修改mac地址 编辑:程序博客网 时间:2024/05/16 05:18
1.关于梯度简单的理解
f(x,y)=xy
可以很容易得到f(x,y)关于x 和y的偏导数
函数关于每个变量的偏导数告诉了你整个函数对于单个变量的敏感程度。
2.链式法则
f(x,y,z)=(x+y)z
可以把上面的公式分解成为
q=x+y 和 f=qz
对y求偏导也同样如此
# set some inputs
x = -2; y = 5; z = -4
写成如下代码
# perform the forward passq = x + y # q becomes 3f = q * z # f becomes -12# perform the backward pass (backpropagation) in reverse order:# first backprop through f = q * zdfdz = q # df/dz = q, so gradient on z becomes 3dfdq = z # df/dq = z, so gradient on q becomes -4# now backprop through q = x + ydfdx = 1.0 * dfdq # dq/dx = 1. And the multiplication here is the chain rule!dfdy = 1.0 * dfdq # dq/dy = 1
2.backpropagation 在sigmoid函数上的求梯度的应用
sigmoid函数如下图所示
想求f对于w 和x 的梯度
下面是除了add mul max 以外最简单的表达式 ,以及相应的导数
下面是sigmoid的导数
下面代码分别对x w求梯度
w = [2,-3,-3] # assume some random weights and datax = [-1, # forward passdot = w[0]*x[0] + w[1]*x[1] + w[2]f = 1.0 / (1 + math.exp(-dot)) # sigmoid funct# backward pass through the neuron (backpropagation)ddot = (1 - f) * f # gradient on dot variable, using the sigmoid gradient derivationdx = [w[0] * ddot, w[1] * ddot] # backprop into xdw = [x[0] * ddot, x[1] * ddot, 1.0 * ddot] # backprop into w# we're done! we have the gradients on the inputs to the circuit
考虑一个更复杂的公式
我们在forward pass 的时候先将这个公式分解成前面部分所阐述的最简单的公式。
代码如下:
x = 3 # example valuesy = -4# forward passsigy = 1.0 / (1 + math.exp(-y)) # sigmoid in numerator #(1)num = x + sigy # numerator #(2)sigx = 1.0 / (1 + math.exp(-x)) # sigmoid in denominator #(3)xpy = x + y #(4)xpysqr = xpy**2 #(5)den = sigx + xpysqr # denominator #(6)invden = 1.0 / den #(7)f = num * invden # done! #(8)
下面的backpropagation 对forward pass 反向求梯度
# backprop f = num * invdendnum = invden # gradient on numerator #(8)dinvden = num #(8)# backprop invden = 1.0 / den dden = (-1.0 / (den**2)) * dinvden #(7)# backprop den = sigx + xpysqrdsigx = (1) * dden #(6)dxpysqr = (1) * dden #(6)# backprop xpysqr = xpy**2dxpy = (2 * xpy) * dxpysqr #(5)# backprop xpy = x + ydx = (1) * dxpy #(4)dy = (1) * dxpy #(4)# backprop sigx = 1.0 / (1 + math.exp(-x))dx += ((1 - sigx) * sigx) * dsigx # Notice += !! See notes below #(3)# backprop num = x + sigydx += (1) * dnum #(2)dsigy = (1) * dnum #(2)# backprop sigy = 1.0 / (1 + math.exp(-y))dy += ((1 - sigy) * sigy) * dsigy #(1)# done! phew
1 0
- Backpropagation
- backpropagation
- Backpropagation
- Backpropagation
- Backpropagation
- backpropagation
- Backpropagation, Intuitions
- BackPropagation算法
- [Machine_Learning]--Backpropagation
- backpropagation algorithm
- backpropagation,intuitions
- Backpropagation Algorithm
- Backpropagation算法
- Resilient Backpropagation
- Machine Learning---Backpropagation
- Sparse Autoencoder2-Backpropagation Algorithm
- Study notes for Backpropagation
- Neural Netowrk : BackPropagation
- java.lang.OutOfMemoryError: PermGen space及其解决方法
- 结合top和jstack找出占用cpu最高的堆栈信息
- FFmpeg 移植到 Android 平台
- C++ primer 读书笔记7.2 ~7.6
- 让App的图标不显示
- backpropagation
- 关于static(静态)变量的一些理解
- wifi质量强度测试
- leetcode【217+219 Contains Duplicate I+II】【python】
- 如何自定义eclipse代码格式化(Ctrl+Shift+F)
- 台式机机箱前耳机插孔无法使用,在控制面版中无法找到Realtek音频管理器
- 重建二叉树
- android调试用到的各种命令
- angularJS自定义指令各配置项详解