A Comprehensive guide to Fine-tuning Deep Learning Models in Keras
来源:互联网 发布:linux如何查看raid 编辑:程序博客网 时间:2024/05/29 06:36
First, why do we fine-tune Models?
When we are given a Deep Learning task, say, one that involves training a Convolutional Neural Network (Covnet) on a dataset of images, our first instinct would be to train the network from scratch. However, in practice, deep neural networks like Covnet has a huge number of parameters, often in the range of millions. Training a Covnet on a small dataset (one that is smaller than the number of parameters) greatly affects the Covnet’s ability to generalize, often result in overfitting.
Therefore, more often in practice, one would fine-tune existing networks that are trained on a large dataset like the ImageNet (1.2M labeled images) by continue training it (i.e. running back-propagation) on the smaller dataset we have. Provided that our dataset is not drastically different in context to the original dataset (e.g. ImageNet), the pre-trained model will already have learned features that are relevant to our own classification problem.
When to fine-tune Models?
In general, if our dataset is not drastically different in context from the dataset which the pre-trained model is trained on, we should go for fine-tuning. Pre-trained network on a large and diverse dataset like the ImageNet captures universal features like curves and edges in its early layers, that are relevant and useful to most of the classification problems.
Of course, if our dataset represents some very specific domain, say for example, medical images or Chinese handwritten characters, and that no pre-trained networks on such domain can be found, we should then consider training the network from scratch.
If our dataset is really small, say less than a thousand samples, a better approach is to take the output of the intermediate layer prior to the fully connected layers as features (bottleneck features) andtrain a linear classifier (e.g. SVM) on top of it. SVM is particularly good at drawing decision boundaries on a small dataset.
Fine-tuning Techniques
Below are some general guidelines for fine-tuning implementation:
1. The common practice is to truncate the last layer (softmax layer) of the pre-trained network and replace it with our new softmax layer that are relevant to our own problem. For example, pre-trained network on ImageNet comes with a softmax layer with 1000 categories.
If our task is a classification on 10 categories, the new softmax layer of the network will be of 10 categories instead of 1000 categories. We then run back propagation on the network to fine-tune the pre-trained weights. Make sure cross validation is performed so that the network will be able to generalize well.
2. Use a smaller learning rate to train the network. Since we expect the pre-trained weights to be quite good already as compared to randomly initialized weights, we do not want to distort them too quickly and too much. A common practice is to make the initial learning rate 10 times smaller than the one used for scratch training.
3. It is also a common practice to freeze the weights of the first few layers of the pre-trained network. This is because the first few layers capture universal features like curves and edges that are also relevant to our new problem.We want to keep those weights intact. Instead, we will get the network to focus on learning dataset-specific features in the subsequent layers.
Original Article:
https://flyyufelix.github.io/2016/10/03/fine-tuning-in-keras-part1.html
- A Comprehensive guide to Fine-tuning Deep Learning Models in Keras
- The E-Learning Handbook: A Comprehensive Guide to Online Learning
- A Guide to Deep Learning
- using learning rate schedules for deep learning models in python with keras
- A guide to convolution arithmetic for deep learning
- A 2017 Guide to Semantic Segmentation with Deep Learning
- Save and Load Your Keras Deep Learning Models
- Keras Tutorial: Deep Learning in Python
- Deep learning系列(十三)Transfer learning 和 caffe Fine-tuning
- transfer learning and fine-tuning
- How to do a deep SQL tuning
- Borrowing treasures from the wealthy: deep transfer learning through selective joint fine-tuning
- Deep Learning in a Nutshell
- A Trip To Deep Learning
- A 2017 Guide to Semantic Segmentation with Deep Learning阅读笔记
- Deep Learning for Content-Based Image Retrival:A Comprehensive Study 学习笔记
- Deep Learning for Content-Based Image Retrival:A Comprehensive Study 论文笔记
- How to Use the Keras Functional API for Deep Learning
- SDUT 3374 数据结构实验之查找二:平衡二叉树
- Linux 关闭tomcat
- Revit二次开发——让窗口一直显示在Revit前面
- C#中的ArrayList集合详解与深度探讨
- centos7 安装后相关配置
- A Comprehensive guide to Fine-tuning Deep Learning Models in Keras
- Delphi整理八(对话框)
- c# enum 枚举一般用法
- js 去掉alert在手机浏览器显示网址的网址信息
- 免费领取地图下载流量与流量使用方法
- 第1章 Python机器学习的生态系统
- 6-2 多态性与虚函数
- 每天一个linux命令(21):find命令之xargs
- Ubuntu16.04安装Shark-3.0.0