Tensorflow实例:实现ResNet
来源:互联网 发布:惊艳句子知乎 编辑:程序博客网 时间:2024/05/29 03:06
ResNet(Residual Neural Network)由微软研究院的Kaiming He等4名华人提出,通过使用Residual Unit成功训练152层深的神经网络,在ILSVRC 2015比赛中获得了冠军,取得3.57%的top-5错误率,同时参数量却比VGGNet低,效果非常突出。
ResNet允许原始输入信息直接传输到后面的层中。ResNet最初的灵感出自这个问题:在不断加神经网络的深度时,会出现一个Degradation的问题,即准确率会先上升然后达到饱和,再持续增加深度则会导致准确率下降。这并不是过拟合的问题,因为不光在测试集上误差增大,训练集本身误差也会增大。假设有一个比较浅的网络达到了饱和的准确率,那么后面再加上几个y=x的全等映射层,起码误差不会增加,即更深的网络不应该带来训练集上误差上升。
Residual Unit
假设某段神经网络的输入是
下图为VGGNet-19、34层深的普通卷积网络、34层深的ResNet网络的对比图:
可以看到,普通直连的卷积神经网络和ResNet的最大区别在于,ResNet有许多旁路的支线将输入直接连到后面的层,使得后面的层可以直接学习残差,这种结构也被称为shortcut或skip connections。
传统的卷积层或全连接层在信息传递时,或多或少会丢失信息、损耗等问题。ResNet在某种程度上解决了这个问题,通过直接将输入信息绕道传到输出,保护信息的完整性,整个网络则只需学习输入、输出差别的那一部分,简化学习目标和难度。
在使用了ResNet的结构后,可以发现层数不断加深导致的训练集上误差增大的现象被消除了,ResNet网络的训练会随着层数增大而逐渐减小,并且在测试集上的表现也会变好。
之后,Residual V2被提出。Residual V2和Residual V1的主要区别在于:
- 作者通过研究ResNet残差学习单元的传播公式,发现前馈和反馈信号可以直接传输,因此skip connection的非线性激活函数(如ReLU)替代为Identity Mapping (y = x)。
ResNet V2在每一层都使用了Batch Normalization。这样处理之后,新的残差学习单元将比以前更容易训练且泛化性更强。
ResNet V2网络的实现
下面我们就用Tensorflow实现一个ResNet V2网络。我们依然使用方便的contrib.slim库来辅助创建ResNet。
import collectionsimport tensorflow as tfslim = tf.contrib.slim# 我们使用collections.namedtuple设计ResNet基本Block模块的named tuple,并用它创建Block的类,# 但只包含数据结构,不包含具体方法。class Block(collections.namedtuple('Block', ['scope', 'unit_fn', 'args'])): 'A named tuple describing a ResNet block'# 下面定义一个降采样subsample的方法def subsample(inputs, factor, scope=None): if factor == 1: return inputs else: return slim.max_pool2d(inputs, [1, 1], stride=factor, scope=scope)# 再定义一个conv2d_same函数创建卷积层。def conv2d_same(inputs, num_outputs, kernel_size, stride, scope=None): if stride == 1: return slim.conv2d(inputs, num_outputs, kernel_size, stride=1, padding='SAME', scope=scope) else: pad_total = kernel_size - 1 pad_beg = pad_total // 2 pad_end = pad_total - pad_beg inputs = tf.pad(inputs, [[0, 0], [pad_beg, pad_end], [pad_beg, pad_end], [0, 0]]) return slim.conv2d(inputs, num_outputs, kernel_size, stride=stride, padding='VALID', scope=scope)# 接下来定义堆叠Blocks的函数,参数中的net即为输入,blocks是之前定义的Block的class的列表,# 而outputs_collections则是用来收集各个end_points的collections。@slim.add_arg_scopedef stack_blocks_dense(net, blocks, outputs_collections=None): for block in blocks: with tf.variable_scope(block.scope, 'block', [net]) as sc: for i, unit in enumerate(block.args): with tf.variable_scope('unit_%d' % (i + 1), values=[net]): unit_depth, unit_depth_bottleneck, unit_stride = unit net = block.unit_fn(net, depth=unit_depth, depth_bottleneck=unit_depth_bottleneck, stride=unit_stride) net = slim.utils.collect_named_outputs(outputs_collections, sc.name, net) return net# 这里创建ResNet通用的arg_scope,用来定义某些函数的参数默认值。def resnet_arg_scope(is_training=True, weight_decay=0.0001, batch_norm_decay=0.997, batch_norm_epsilon=1e-5, batch_norm_scale=True): batch_norm_scale = { 'is_training': is_training, 'decay': batch_norm_decay, 'epsilon': batch_norm_epsilon, 'scale': batch_norm_scale, 'updates_collections': tf.GraphKeys.UPDATE_OPS, } with slim.arg_scope( [slim.conv2d], weights_regularizer = slim.l2_regularizer(weight_decay), weights_initializer=slim.variance_scaling_initializer(), activation_fn = tf.nn.relu, normalizer_fn = slim.batch_norm, normalizer_params = batch_norm_parms ): with slim.arg_scope([slim.batch_norm], **batch_norm_params): with slim.arg_scope([slim.max_pool2d], padding='SAME') as arg_sc: return arg_sc# 接下来定义核心的bottlneck残差学习单元@slim.add_arg_scopedef bottleneck(inputs, depth, depth_bottleneck, stride, outputs_collections=None, scope=None): with tf.variable_scope(scope, 'bottleneck_v2', [inputs]) as sc: depth_in = slim.utils.last_dimension(inputs.get_shape(), min_rank=4) preact = slim.batch_norm(inputs, activation_fn=tf.nn.relu, scope='preact') if depth == depth_in: shortcut = subsample(inputs, stride, 'shortcut') else: shortcut = slim.conv2d(preact, depth, [1, 1], stride=stride, normalizer_fn=None, activation_fn=None, scope='shortcut') residual = slim.conv2d(preact, depth_bottleneck, [1, 1], stride=1, scope='conv1') residual = conv2d_same(residual, depth_bottleneck, 3, stride, scope='conv2') residual = slim.conv2d(residual, depth, [1, 1], stride=1, normalizer_fn=None, activation_fn=None, scope='conv3') output = shortcut + residual return slim.utils.collect_named_outputs(outputs_collections, sc.name, output)# 下面定义生成ResNet V2的主函数,我们只要预先定义好网络的残差学习模块组blocks,它就可以生成对应# 的完整的ResNetdef resnet_v2(inputs, blocks, num_classes=None, global_pool=True, include_root_block=True, reuse=None, scope=None): with tf.variable_scope(scope, 'resnet_v2', [inputs], reuse=reuse) as sc: end_point_collection = sc.original_name_scope + '_end_points' with slim.arg_scope([slim.conv2d, bottleneck, stack_blocks_dense], outputs_collections=end_point_collection): net = inputs if include_root_block: with slim.arg_scope([slim.conv2d], activation_fn=None, normalizer_fn=None): net = conv2d_same(net,64, 7, stride=2, scope='conv1') net = slim.max_pool2d(net, [3, 3], stride=2, scope='pool1') net = stack_blocks_dense(net, blocks) net = slim.batch_norm(net, activation_fn=tf.nn.relu, scope='postnorm') if global_pool: net = tf.reduce_mean(net, [1, 2], name='pool5', keep_dims=True) if num_classes is not None: net = slim.conv2d(net, num_classes, [1, 1], activation_fn=None, normalizer_fn=None, scope='logits') end_points = slim.utils.convert_collection_to_dict(end_point_collection) if num_classes is not None: end_points['predictions'] = slim.softmax(net, scope='predictions') return net, end_points# 至此,我们就将ResNet的生成函数定义好了。下面根据几个不同深度的ResNet网络配置,来设计层数# 分别为50, 101, 152, 200的ResNet。def resnet_v2_50(inputs, num_classes=None, global_pool=True, reuse=None, scope='resnet_v2_50'): blocks = [ Block('block1', bottleneck, [(256, 64, 1)] * 2 + [(256, 64, 2)]), Block('block2', bottleneck, [(512, 128, 1)] * 3 + [(512, 128, 2)]), Block('block3', bottleneck, [(1024, 256, 1)] * 5 + [(1024, 256, 2)]), Block('block4', bottleneck, [(2048, 512, 1)] * 3)] return resnet_v2(inputs, blocks, num_classes, global_pool, include_root_block=True, reuse=reuse, scope=scope)def resnet_v2_101(inputs, num_classes=None, global_pool=True, reuse=None, scope='resnet_v2_101'): blocks = [ Block('block1', bottleneck, [(256, 64, 1)] * 2 + [(256, 64, 2)]), Block('block2', bottleneck, [(512, 128, 1)] * 3 + [(512, 128, 2)]), Block('block3', bottleneck, [(1024, 256, 1)] * 22 + [(1024, 256, 2)]), Block('block4', bottleneck, [(2048, 512, 1)] * 3) ] return resnet_v2(inputs, blocks, num_classes, global_pool, include_root_block=True, reuse=reuse, scope=scope)def resnet_v2_152(inputs, num_classes=None, global_pool=True, reuse=None, scope='resnet_v2_152'): blocks = [ Block('block1', bottleneck, [(256, 64, 1)] * 2 + [(256, 64, 2)]), Block('block2', bottleneck, [(512, 128, 1)] * 7 + [(512, 128, 2)]), Block('block3', bottleneck, [(1024, 256, 1)] * 35 + [(1024, 256, 2)]), Block('block4', bottleneck, [(2048, 512, 1)] * 3) ] return resnet_v2(inputs, blocks, num_classes, global_pool, include_root_block=True, reuse=reuse, scope=scope)def resnet_v2_200(inputs, num_classes=None, global_pool=True, reuse=None, scope='resnet_v2_200'): blocks = [ Block('block1', bottleneck, [(256, 64, 1)] * 2 + [(256, 64, 2)]), Block('block2', bottleneck, [(512, 128, 1)] * 23 + [(512, 128, 2)]), Block('block3', bottleneck, [(1024, 256, 1)] * 35 + [(1024, 256, 2)]), Block('block4', bottleneck, [(2048, 512, 1)] * 3) ] return resnet_v2(inputs, blocks, num_classes, global_pool, include_root_block=True, reuse=reuse, scope=scope)batch_size = 32height, width = 224, 224inputs = tf.random_uniform((batch_size, height, width, 3))with slim.arg_scope(resnet_arg_scope(is_training=False)): net, end_points = resnet_v2_152(inputs, 1000)init = tf.global_variables_initializer()sess = tf.Session()sess.run(init)num_batches = 100time_tensorflow_run(sess, net, "Forward")
- Tensorflow实例:实现ResNet
- Resnet的Tensorflow实现源代码
- Tensorflow实战学习(三十三)【实现ResNet】
- TensorFlow学习--ResNet实现
- tensorflow36《TensorFlow实战》笔记-06-04 TensorFlow实现 ResNet code
- TensorFlow实现ResNet(ResNet 152网络结构的forward耗时检测)
- TensorFlow实战10:ResNet实现及时间测评
- 神经网络之ResNet模型的实现(Python+TensorFlow)
- ResNet实现
- TensorFlow实现经典深度学习网络(4):TensorFlow实现ResNet
- Tensorflow实例:实现AlexNet
- Tensorflow实例:实现VGG
- tensorflow实现迁移学习实例
- ResNet-TensorFlow Model Zoo代码理解
- 理解ResNet结构与TensorFlow代码分析
- 理解ResNet结构与TensorFlow代码分析
- 理解ResNet结构与TensorFlow代码分析
- Tensorflow slim resnet v2源码阅读笔记
- 唐纳德与子串 (Easy)(计算子串个数)
- bzoj4196 [Noi2015]软件包管理器 树链剖分
- [BZOJ1412] [ZJOI2009] 狼与羊的故事 (最小割)
- 1062. 最简分数(20)
- PHP上传文件$_FILES, $_POST为空 empty 时, 文件上传大小限制
- Tensorflow实例:实现ResNet
- 豆豆学python——(2)字符串和编码
- 第九周项目3
- 大数据-开发利器
- python 笔记
- Slim 框架学习,第八天 _Container(五)
- 637. Average of Levels in Binary Tree
- NYOJ 21 三个水杯
- Access restriction: The type ‘BASE64Decoder’ is not API (restriction on required library ‘D:\java\jd