优化算法动画演示Alec Radford's animations for optimization algorithms
来源:互联网 发布:简易灶台橱柜淘宝店铺 编辑:程序博客网 时间:2024/05/06 22:12
Alec Radford has created some great animations comparing optimization algorithms SGD, Momentum, NAG, Adagrad, Adadelta,RMSprop (unfortunately no Adam) on low dimensional problems. Also check out his presentation on RNNs.
"Noisy moons: This is logistic regression on noisy moons dataset from sklearn which shows the smoothing effects of momentum based techniques (which also results in over shooting and correction). The error surface is visualized as an average over the whole dataset empirically, but the trajectories show the dynamics of minibatches on noisy data. The bottom chart is an accuracy plot."
"Beale's function: Due to the large initial gradient, velocity based techniques shoot off and bounce around - adagrad almost goes unstable for the same reason. Algos that scale gradients/step sizes like adadelta and RMSProp proceed more like accelerated SGD and handle large gradients with more stability."
"Long valley: Algos without scaling based on gradient information really struggle to break symmetry here - SGD gets no where and Nesterov Accelerated Gradient / Momentum exhibits oscillations until they build up velocity in the optimization direction. Algos that scale step size based on the gradient quickly break symmetry and begin descent."
"Saddle point: Behavior around a saddle point. NAG/Momentum again like to explore around, almost taking a different path. Adadelta/Adagrad/RMSProp proceed like accelerated SGD."
from: http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html
0 0
- 优化算法动画演示Alec Radford's animations for optimization algorithms
- Andrew Ng's deeplearning Course2Week2 Optimization algorithms(优化算法)
- Numerical algorithms for nonsmooth optimization
- 深度神经网络优化(二)- Optimization algorithms
- 梯度下降算法优化综述An overview of gradient descent optimization algorithms
- Coursera deeplearning.ai 深度学习笔记2-2-Optimization algorithms-优化算法与代码实现
- Optimization Algorithms
- Euclid's algorithms for GCD
- 动画Animations
- SZU:B85 Alec's Eggs
- MLlib - Optimization-based Algorithms
- Proximal Policy Optimization Algorithms
- SMO优化算法(Sequential minimal optimization)
- SMO优化算法(Sequential minimal optimization)
- SMO优化算法(Sequential minimal optimization)
- 算法中的优化问题(optimization problem)
- SMO优化算法(Sequential minimal optimization)
- win32下for循环优化(Optimization on for loop)
- U3D HTTP 最好用的插件BestHttp
- 计算机程序的构造和解释练习题
- 如何写科技论文How to write a technical paper
- 项目遇到的小问题
- [C/C++]_[判断程序是32位还是64位]
- 优化算法动画演示Alec Radford's animations for optimization algorithms
- 飛飛(十三)求四个数的最大公约数
- 75. Sort Colors
- 要不要进入互联网行业
- mysql主从手动切换
- TableTools导出的文件csv都是中文乱码解决办法
- TCP/UDP协议常用端口号服务
- Ubuntu 14.04 LTS 64位安装Oracle 11g (一)
- 让vim记住之前打开文件的光标位置