2016.4.15 近期要读的论文

图像识别

1. Krizhevsky, A., Sutskever, I. & Hinton, G. ImageNetclassification with deepconvolutional neural networks. In Proc. Advances inNeural InformationProcessing Systems 25 1090–1098 (2012).This report was abreakthrough that used convolutional nets to almost halvethe error rate forobject recognition, and precipitated the rapid adoption ofdeep learning by thecomputer vision community.

2. Farabet, C., Couprie, C., Najman, L. & LeCun, Y. Learninghierarchical features forscene labeling. IEEE Trans. Pattern Anal. Mach.Intell. 35, 1915–1929 (2013).

3. Tompson, J., Jain, A., LeCun, Y. & Bregler, C. Joint trainingof a convolutionalnetwork and a graphical model for human pose estimation. InProc. Advances inNeural Information Processing Systems 27 1799–1807 (2014).

4. Szegedy, C. et al. Going deeper with convolutions. Preprint athttp://arxiv.org/abs/1409.4842 (2014).

使用ReLU从而避免unsupervisedpre-training

28. Glorot, X., Bordes, A. & Bengio. Y. Deep sparse rectifierneural networks. In Proc.14th International Conference on ArtificialIntelligence and Statistics 315–323(2011).

This paper showed thatsupervised training of very deep neural networks is much faster if the hiddenlayers are composed of ReLU.

深度网络重燃战火

31. Hinton, G. E. What kind of graphical model is the brain? InProc. 19^th International Joint Conference on Artificial intelligence1765–1775 (2005).

32. Hinton, G. E., Osindero, S. & Teh, Y.-W. A fast learningalgorithm for deep belief nets. Neural Comp. 18, 1527–1554 (2006).

This paper introduced anovel and effective way of training very deep neural networks by pre-trainingone hidden layer at a time using the unsupervised learning procedure forrestricted Boltzmann machines.

33. Bengio, Y., Lamblin, P., Popovici, D. & Larochelle, H.Greedy layer-wise training of deep networks. In Proc. Advances in NeuralInformation Processing Systems 19 153–160 (2006).

This report demonstratedthat the unsupervised pre-training method introduced in ref. 32 significantlyimproves performance on test data and generalizes the method to otherunsupervised representation-learning techniques, such as auto-encoders.

34. Ranzato, M., Poultney, C., Chopra, S. & LeCun, Y. Efficientlearning of sparse representations with an energy-based model. In Proc.Advances in Neural Information Processing Systems 19 1137–1144 (2006).

无监督初始化，bp fine-tune

33. Bengio, Y., Lamblin, P., Popovici, D. & Larochelle, H.Greedy layer-wise trainingof deep networks. In Proc. Advances in NeuralInformation Processing Systems 19 153–160 (2006).

This report demonstratedthat the unsupervised pre-training method introduced in ref. 32 significantlyimproves performance on test data and generalizes the method to otherunsupervised representation-learning techniques, such as auto-encoders.

34. Ranzato, M., Poultney, C., Chopra, S. & LeCun, Y. Efficientlearning of sparse representations with an energy-based model. In Proc.Advances in Neural Information Processing Systems 19 1137–1144 (2006).

35. Hinton, G. E. & Salakhutdinov, R. Reducing thedimensionality of data with neural networks. Science 313, 504–507 (2006).

小数据上采用pre-training + fine-tune进行手写数字识别和行人检测

36. Sermanet, P., Kavukcuoglu, K., Chintala, S. & LeCun, Y.Pedestrian detection with unsupervised multi-stage feature learning. In Proc.International Conference on Computer Vision and Pattern Recognitionhttp://arxiv.org/abs/1212.0142 (2013).

采用gpu进行训练

37. Raina, R., Madhavan, A. & Ng, A. Y. Large-scale deepunsupervised learning using graphics processors. In Proc. 26th AnnualInternational Conference on Machine Learning 873–880 (2009).

小数据集上pre-training 防止过拟合

40. Bengio, Y., Courville, A. & Vincent, P. Representationlearning: a review and new perspectives. IEEE Trans. Pattern Anal. MachineIntell. 35, 1798–1828 (2013).

卷积神经网络

41. LeCun, Y. et al. Handwritten digit recognition with aback-propagation network. In Proc. Advances in Neural Information ProcessingSystems 396–404 (1990).

This is the first paper onconvolutional networks trained by backpropagation for the task of classifyinglow-resolution images of handwritten digits.

42. LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P.Gradient-based learning applied to document recognition. Proc. IEEE 86,2278–2324 (1998).

This overview paper on theprinciples of end-to-end training of modular systems such as deep neuralnetworks using gradient-based optimization showed how neural networks (and inparticular convolutional nets) can be combined with search or inferencemechanisms to model complex outputs that are interdependent, such as sequencesof characters associated with the content of a document.

视觉神经元启发卷积和池化层

43. Hubel, D. H. & Wiesel, T. N. Receptive fields, binocularinteraction, and functional architecture in the cat’s visual cortex. J.Physiol. 160, 106–154 (1962).

44. Felleman, D. J. & Essen, D. C. V. Distributed hierarchicalprocessing in the primate cerebral cortex. Cereb. Cortex 1, 1–47 (1991).

一个研究关于convnet和猴子面对同一个神经元在高层次的表现

45. Cadieu, C. F. et al. Deep neural networks rival therepresentation of primate it cortex for core visual object recognition. PLoSComp. Biol. 10, e1003963 (2014).

微软进行光学字符识别和手写数字识别

49. Simard, D., Steinkraus, P. Y. & Platt, J. C. Best practicesfor convolutional neural networks. In Proc. Document Analysis and Recognition958–963 (2003).

自然图片中的物体检测

50. Vaillant, R., Monrocq, C. & LeCun, Y. Original approach forthe localisation of objects in images. In Proc. Vision, Image, and SignalProcessing 141, 245–250(1994).

51. Nowlan, S. & Platt, J. in Neural Information ProcessingSystems 901–908 (1995).

面部识别

52. Lawrence, S., Giles, C. L., Tsoi, A. C. & Back, A. D. Facerecognition: a convolutional neural-network approach. IEEE Trans. NeuralNetworks 8, 98–113(1997).

交通信号识别

53. Ciresan, D.,Meier, U. Masci, J. & Schmidhuber, J. Multi-column deep neural network fortraffic sign classification. Neural Networks 32, 333–338 (2012).

生物图像切割

54. Ning, F. etal. Toward automatic phenotyping of developing embryos from videos. IEEE Trans.Image Process. 14, 1360–1371 (2005).

面部检测、行人检测、躯干检测等

36. Sermanet, P., Kavukcuoglu, K., Chintala, S. & LeCun, Y.Pedestrian detection with unsupervised multi-stage feature learning. In Proc.International Conference on Computer Vision and Pattern Recognitionhttp://arxiv.org/abs/1212.0142 (2013).

50. Vaillant, R., Monrocq, C. & LeCun, Y. Original approach forthe localisation of objects in images. In Proc. Vision, Image, and SignalProcessing 141, 245–250(1994).

51. Nowlan, S. & Platt, J. in Neural Information ProcessingSystems 901–908 (1995).

56. Garcia, C.& Delakis, M. Convolutional face finder: a neural architecture for fast androbust face detection. IEEE Trans. Pattern Anal. Machine Intell. 26,1408–1423(2004).

57. Osadchy, M.,LeCun, Y. & Miller, M. Synergistic face detection and pose estimation withenergy-based models. J. Mach. Learn. Res. 8, 1197–1215 (2007).

58. Tompson, J.,Goroshin, R. R., Jain, A., LeCun, Y. Y. & Bregler, C. C. Efficient object localizationusing convolutional networks. In Proc. Conference on Computer Vision andPattern Recognition http://arxiv.org/abs/1411.4280(2014).

面部识别

59. Taigman, Y.,Yang, M., Ranzato, M. & Wolf, L. Deepface: closing the gap to human-levelperformance in face verification. In Proc. Conference on Computer Vision andPattern Recognition 1701–1708 (2014).

dropout

62. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. &Salakhutdinov, R. Dropout: a simple way to prevent neural networks fromoverfitting. J. Machine Learning Res. 15, 1929–1958 (2014).

识别和检测

4. Szegedy, C. et al. Going deeper with convolutions. Preprint athttp://arxiv.org/abs/1409.4842 (2014).

58. Tompson, J., Goroshin, R. R., Jain, A., LeCun, Y. Y. &Bregler, C. C. Efficient object localization using convolutional networks. InProc. Conference on Computer Vision and Pattern Recognitionhttp://arxiv.org/abs/1411.4280 (2014).

59. Taigman, Y., Yang, M., Ranzato, M. & Wolf, L. Deepface:closing the gap to human-level performance in face verification. In Proc.Conference on Computer Vision and Pattern Recognition 1701–1708 (2014).

63. Sermanet, P. et al. Overfeat: integrated recognition,localization and detection using convolutional networks. In Proc. InternationalConference on Learning Representations http://arxiv.org/abs/1312.6229 (2014).

64. Girshick, R., Donahue, J., Darrell, T. & Malik, J. Richfeature hierarchies for accurate object detection and semantic segmentation. InProc. Conference on Computer Vision and Pattern Recognition 580–587 (2014).

65. Simonyan, K. & Zisserman, A. Very deep convolutionalnetworks for large-scale image recognition. In Proc. International Conferenceon Learning Representations http://arxiv.org/abs/1409.1556 (2014).

distributedrepresentations

21. Bengio, Y., Delalleau, O. & Le Roux, N. The curse of highlyvariable functions for local kernel machines. In Proc. Advances in NeuralInformation Processing Systems 18 107–114 (2005).

数据分布下的整体架构

40. Bengio, Y., Courville, A. & Vincent, P. Representationlearning: a review and new perspectives. IEEE Trans. Pattern Anal. MachineIntell. 35, 1798–1828 (2013).

distributedrepresentations增强泛化能力

68. Bengio, Y. Learning Deep Architectures for AI (Now, 2009).

69. Montufar, G. & Morton, J. When does a mixture of productscontain a product of mixtures? J. Discrete Math. 29, 321–347 (2014).

深度增强表达能力

70. Montufar, G. F., Pascanu, R., Cho, K. & Bengio, Y. On thenumber of linear regions of deep neural networks. In Proc. Advances in NeuralInformation Processing Systems 27 2924–2932 (2014).

通过局部输入确定下一个输出

71. Bengio, Y., Ducharme, R. & Vincent, P. A neuralprobabilistic language model. In Proc. Advances in Neural InformationProcessing Systems 13 932–938 (2001). This paper introduced neural languagemodels, which learn to convert a word symbol into a word vector or wordembedding composed of learned semantic features in order to predict the nextword in a sequence.

非监督学习

91. Hinton, G. E., Dayan, P., Frey, B. J. & Neal, R. M. Thewake-sleep algorithm for unsupervised neural networks. Science 268, 1558–1161(1995).

92. Salakhutdinov, R. & Hinton, G. Deep Boltzmann machines. InProc. International Conference on Artificial Intelligence and Statistics448–455 (2009).

93. Vincent, P., Larochelle, H., Bengio, Y. & Manzagol, P.-A.Extracting and composing robust features with denoising autoencoders. In Proc.25th International Conference on Machine Learning 1096–1103 (2008).

94. Kavukcuoglu, K. et al. Learning convolutional featurehierarchies for visual recognition. In Proc. Advances in Neural InformationProcessing Systems 23 1090–1098 (2010).

95. Gregor, K. & LeCun, Y. Learning fast approximations ofsparse coding. In Proc. International Conference on Machine Learning 399–406(2010).

96. Ranzato, M., Mnih, V., Susskind, J. M. & Hinton, G. E.Modeling natural images using gated MRFs. IEEE Trans. Pattern Anal. MachineIntell. 35, 2206–2222(2013).

97. Bengio, Y., Thibodeau-Laufer, E., Alain, G. & Yosinski, J.Deep generative stochastic networks trainable by backprop. In Proc. 31stInternational Conference on Machine Learning 226–234 (2014).

98. Kingma, D., Rezende, D., Mohamed, S. & Welling, M.Semi-supervised learning with deep generative models. In Proc. Advances inNeural Information Processing Systems 27 3581–3589 (2014).

cnn＋rnn使用增强学习进行视觉分类

99. Ba, J., Mnih, V. & Kavukcuoglu, K. Multiple objectrecognition with visual attention. In Proc. International Conference onLearning Representations。http://arxiv.org/abs/1412.7755 (2014).

cnn＋rnn使用增强学习玩游戏

100. Mnih, V. et al. Human-level control through deep reinforcementlearning. Nature518, 529–533 (2015).

rnn learn strategies forselectively attending to one part at a time

76. Bahdanau, D., Cho, K. & Bengio, Y. Neural machinetranslation by jointly learning to align and translate. In Proc. InternationalConference on Learning Representations http://arxiv.org/abs/1409.0473 (2015).

86. Xu, K. et al. Show, attend and tell: Neural image captiongeneration with visual attention. In Proc. International Conference on LearningRepresentations http://arxiv.org/abs/1502.03044 (2015).

rnn关注图片特定位置

102. Vinyals, O., Toshev, A., Bengio, S. & Erhan, D. Show andtell: a neural image caption generator. In Proc. International Conference onMachine Learning http://arxiv.org/abs/1502.03044 (2014).