Chapter 1 - Introduction - 深度学习的来历 - 三次大潮

• In fact, deep learning dates back to the 1940s. Deep
  learning only appears to be new,

• Broadly speaking, there have been three waves of development of deep learning: deep learning known as cybernetics in the 1940s–1960s, deeplearning known as connectionism in the 1980s–1990s, and the current resurgence under the name deep learning beginning in 2006.

• While the kinds of neural networks used for machine learning have sometimes been used to understand brain function, they are generally not designed to be realistic models of biological function.

• Today, neuroscience is regarded as an important source of inspiration for deep learning researchers, but it is no longer the predominant guide for the field.

• The main reason for the diminished role of neuroscience in deep learning
  research today is that we simply do not have enough information about the brain to use it as a guide.

Cybernetics

• f(x,w)=x1w1+⋅⋅⋅xnwn – ~1940
• The adaptive linear element (ADALINE), which affected SGD. – Widrow and Hoff 1960

• Linear models. Linear models have many limitations. Most famously, they cannot learn the XOR function.

• Neuroscience has given us a reason to hope that a single deep learning algorithm can solve many different tasks.

  • Neuroscientists have found that ferrets can learn to “see” with the auditory processing region of their brain if their brains are rewired to send visual signals to that area.

• While neuroscience is an important source of inspiration, it need not be taken as a rigid guide.

  • One should not view deep learning as an attempt
    to simulate the brain.

Connectionism(parallel distributed processor )

• The central idea in connectionism is that a large number of simple computational units can achieve intelligent behavior when networked together.

  • distributed representation
  • popularization of the back-propagation algorithm

• The second wave of neural networks research lasted until the mid-1990s. Ventures based on neural networks and other AI technologies began to make unrealistically ambitious claims while seeking investments.

  • When AI research did not fulfill these unreasonable expectations, investors were disappointed.
  • Other fields of machine learning made advances. e.g. kernel machines, graphic models.

Deep learning

• Deep belief network – Geoffrey Hinton, 2006

• This wave of neural networks research popularized the use of the term deep learning to emphasize that researchers were now able to train deeper neural networks than had been possible before, and to focus attention on the theoretical importance of depth.

• At this time, deep neural networks outperformed competing AI systems based on other machine learning technologies as well as hand-designed functionality.