引言

本文主要用于对7个经典卷积神经网络的初识，大致了解各个网络提出的背景，以及各自对卷积神经网络发展的作用（即网络的特点）。

• 经典的卷积神经网络：
  • LeNet
  • AlexNet
  • ZF Net
  • VGG
  • GoogLeNet
  • ResNet
  • DenseNet

参考： CNN网络架构演进：从LeNet到DenseNet（本文内容来源，强力推荐）

LeNet（5层）

• 提出背景：LeCun在1998年提出，用于解决手写数字识别的视觉任务。

• 历史意义：定义了CNN的基本组件，是CNN的鼻祖。自此，CNN的最基本的架构确定下来：卷积层、池化层、全连接层。如今各大深度学习框架中所使用的LeNet都是简化改进过的LeNet-5（-5表示具有5个层），和原始的LeNet有些许不同，比如把激活函数改为了现在很常用的ReLu。

• 经典的LeNet-5网络结构：如下图。

  

• 数据矩阵计算：
  • 输入是单通道的28*28大小矩阵，用矩阵表示就是[1,28,28]；
  • cov1：20个5*5的卷积核，滑动步长为1，经过该层后尺寸变为24，28-5+1=24，输出矩阵为[20,24,24]；
  • pool1：2*2，步长2，池化操作后，尺寸减半，变为12×12，输出矩阵为[20,12,12]；
  • cov2：50个5*5的卷积核，步长1，卷积后尺寸变为8,12-5+1=8，输出矩阵为[50,8,8]；
  • pool2：2*2，步长2，池化操作后，尺寸减半，变为4×4，输出矩阵为[50,4,4]；
  • fc1：输入为[50,4,4]，神经元数目为500，再接relu激活函数，输出为500；
  • fc2：输入为500，神经元个数为10，得到10维的特征向量；
  • output：输入为10维的特征向量，送入softmaxt分类，得到分类结果的概率output。

    cov为卷积层，pool为池化层，fc的全连接层。

AlexNet（8层）

• 提出背景：AlexNet在2012年ImageNet竞赛中以超过第二名10.9个百分点的绝对优势一举夺冠，从此深度学习和卷积神经网络名声鹊起，深度学习的研究如雨后春笋般出现，AlexNet的出现可谓是卷积神经网络的王者归来。

• 历史意义：

  • 更深的网络
  • 数据增广：数据增广技巧来增加模型泛化能力；
  • ReLU：ReLU代替Sigmoid来加快SGD的收敛速度；
  • dropout：有效缓解了模型的过拟合；
  • LRN：局部响应归一化，减小高激活神经元的使用，后发现这个不起作用，目前已弃用。

Dropout原理类似于浅层学习算法的中集成算法，该方法通过让全连接层的神经元（该模型在前两个全连接层引入Dropout）以一定的概率失去活性（比如0.5）失活的神经元不再参与前向和反向传播，相当于约有一半的神经元不再起作用。在测试的时候，让所有神经元的输出乘0.5。Dropout的引用，有效缓解了模型的过拟合。

Local Responce Normalization（LRN）：局部响应归一层的基本思路是，假如这是网络的一块，比如是 13×13×256， LRN 要做的就是选取一个位置，比如说这样一个位置，从这个位置穿过整个通道，能得到 256 个数字，并进行归一化。进行局部响应归一化的动机是，对于这张 13×13 的图像中的每个位置来说，我们可能并不需要太多的高激活神经元。但是后来，很多研究者发现 LRN 起不到太大作用，因为并不重要，而且我们现在并不用 LRN 来训练网络。

ZF-Net（8层）

• 提出背景：ZFNet是2013ImageNet分类任务的冠军，其网络结构没什么改进，主要是参数的变化，性能较Alex提升了不少。ZF-Net只是将AlexNet第一层卷积核由11变成7，步长由4变为2，第3，4，5卷积层转变为384，384，256。

VGG-Nets（19层）

• 提出背景：VGG-Nets是由牛津大学VGG（Visual Geometry Group）提出，是2014年ImageNet竞赛定位任务的第一名和分类任务的第二名的中的基础网络。VGG可以看成是加深版本的AlexNet. 都是conv layer + FC layer，在当时看来这是一个非常深的网络了，因为层数高达十多层，我们从其论文名字就知道了（《Very Deep Convolutional Networks for Large-Scale Visual Recognition》），当然以现在的目光看来VGG真的称不上是一个very deep的网络。

• 历史意义：

  • 卷积层使用更小的filter尺寸和间隔：VGG-Nets，用到的卷积核的尺寸无非都是1×1和3×3的小卷积核，可以替代大的filter尺寸；

3×3卷积核的优点：

• 多个3×3的卷基层比一个大尺寸filter卷基层有更多的非线性，使得判决函数更加具有判决性
• 多个3×3的卷积层比一个大尺寸的filter有更少的参数，假设卷基层的输入和输出的特征图大小相同为C，那么三个3×3的卷积层参数个数3×（3×3×C×C）=27CC；一个7×7的卷积层参数为49CC；所以可以把三个3×3的filter看成是一个7×7filter的分解（中间层有非线性的分解）

1*1卷积核的优点：作用是在不影响输入输出维数的情况下，对输入进行线性形变，然后通过Relu进行非线性处理，增加网络的非线性表达能力。

GoogLeNet（22层）

• 提出背景：GoogLeNet在2014的ImageNet分类任务上击败了VGG-Nets夺得冠军，其实力肯定是非常深厚的，GoogLeNet跟AlexNet,VGG-Nets这种单纯依靠加深网络结构进而改进网络性能的思路不一样，它另辟幽径，在加深网络的同时（22层），也在网络结构上做了创新，引入Inception结构代替了单纯的卷积+激活的传统操作（这思路最早由Network in Network提出）。GoogLeNet进一步把对卷积神经网络的研究推上新的高度。

• 历史意义：
  • 引入Inception结构；
  • 中间层的辅助LOSS单元；
  • 后面的全连接层全部替换为简单的全局平均pooling。

ResNet（152层）

• 提出背景：2015年何恺明推出的ResNet在ISLVRC和COCO上横扫所有选手，获得冠军。ResNet在网络结构上做了大创新，而不再是简单的堆积层数，ResNet在卷积神经网络的新思路，绝对是深度学习发展历程上里程碑式的事件。

• 历史意义：

  • 层数非常深，已经超过百层；
  • 引入残差单元来解决退化问题；

• 维度匹配方案：

  • zero_padding:对恒等层进行0填充的方式将维度补充完整，这种方法不会增加额外的参数；
  • projection:在恒等层采用1x1的卷积核来增加维度。这种方法会增加额外的参数。

DenseNet

• 提出背景：自Resnet提出以后，ResNet的变种网络层出不穷，都各有其特点，网络性能也有一定的提升。本文介绍的最后一个网络是CVPR 2017最佳论文DenseNet，论文中提出的DenseNet（Dense Convolutional Network）主要还是和ResNet及Inception网络做对比，思想上有借鉴，但却是全新的结构，网络结构并不复杂，却非常有效，在CIFAR指标上全面超越ResNet。可以说DenseNet吸收了ResNet最精华的部分，并在此上做了更加创新的工作，使得网络性能进一步提升。

• 历史意义：密集连接：缓解梯度消失问题，加强特征传播，鼓励特征复用，极大的减少了参数量

在同层深度下获得更好的收敛率，DenseNet具有非常强大的内存占用。

keras代码实现：

• LeNet的Keras实现：

  def LeNet():
      model = Sequential()
      model.add(Conv2D(32,(5,5),strides=(1,1),input_shape=(28,28,1),padding='valid',activation='relu',kernel_initializer='uniform'))
      model.add(MaxPooling2D(pool_size=(2,2)))
      model.add(Conv2D(64,(5,5),strides=(1,1),padding='valid',activation='relu',kernel_initializer='uniform'))
      model.add(MaxPooling2D(pool_size=(2,2)))
      model.add(Flatten())
      model.add(Dense(100,activation='relu'))
      model.add(Dense(10,activation='softmax'))
      return model

• AlexNet的Keras实现：

  def AlexNet():
  
      model = Sequential()
      model.add(Conv2D(96,(11,11),strides=(4,4),input_shape=(227,227,3),padding='valid',activation='relu',kernel_initializer='uniform'))
      model.add(MaxPooling2D(pool_size=(3,3),strides=(2,2)))
      model.add(Conv2D(256,(5,5),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(MaxPooling2D(pool_size=(3,3),strides=(2,2)))
      model.add(Conv2D(384,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(Conv2D(384,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(Conv2D(256,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(MaxPooling2D(pool_size=(3,3),strides=(2,2)))
      model.add(Flatten())
      model.add(Dense(4096,activation='relu'))
      model.add(Dropout(0.5))
      model.add(Dense(4096,activation='relu'))
      model.add(Dropout(0.5))
      model.add(Dense(1000,activation='softmax'))
      return model

• ZF-Net的Keras实现：

  def ZF_Net():
      model = Sequential()  
      model.add(Conv2D(96,(7,7),strides=(2,2),input_shape=(224,224,3),padding='valid',activation='relu',kernel_initializer='uniform'))  
      model.add(MaxPooling2D(pool_size=(3,3),strides=(2,2)))  
      model.add(Conv2D(256,(5,5),strides=(2,2),padding='same',activation='relu',kernel_initializer='uniform'))  
      model.add(MaxPooling2D(pool_size=(3,3),strides=(2,2)))  
      model.add(Conv2D(384,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))  
      model.add(Conv2D(384,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))  
      model.add(Conv2D(256,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))  
      model.add(MaxPooling2D(pool_size=(3,3),strides=(2,2)))  
      model.add(Flatten())  
      model.add(Dense(4096,activation='relu'))  
      model.add(Dropout(0.5))  
      model.add(Dense(4096,activation='relu'))  
      model.add(Dropout(0.5))  
      model.add(Dense(1000,activation='softmax'))  
      return model

• VGG-16的Keras实现：

  def VGG_16():   
      model = Sequential()
      
      model.add(Conv2D(64,(3,3),strides=(1,1),input_shape=(224,224,3),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(Conv2D(64,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(MaxPooling2D(pool_size=(2,2)))
      
      model.add(Conv2D(128,(3,2),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(Conv2D(128,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(MaxPooling2D(pool_size=(2,2)))
      
      model.add(Conv2D(256,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(Conv2D(256,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(Conv2D(256,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(MaxPooling2D(pool_size=(2,2)))
      
      model.add(Conv2D(512,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(Conv2D(512,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(Conv2D(512,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(MaxPooling2D(pool_size=(2,2)))
      
      model.add(Conv2D(512,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(Conv2D(512,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(Conv2D(512,(3,3),strides=(1,1),padding='same',activation='relu',kernel_initializer='uniform'))
      model.add(MaxPooling2D(pool_size=(2,2)))
      
      model.add(Flatten())
      model.add(Dense(4096,activation='relu'))
      model.add(Dropout(0.5))
      model.add(Dense(4096,activation='relu'))
      model.add(Dropout(0.5))
      model.add(Dense(1000,activation='softmax'))
      
      return model

• GoogLeNet的Keras实现：

  def Conv2d_BN(x, nb_filter,kernel_size, padding='same',strides=(1,1),name=None):
      if name is not None:
          bn_name = name + '_bn'
          conv_name = name + '_conv'
      else:
          bn_name = None
          conv_name = None
  
      x = Conv2D(nb_filter,kernel_size,padding=padding,strides=strides,activation='relu',name=conv_name)(x)
      x = BatchNormalization(axis=3,name=bn_name)(x)
      return x
  
  def Inception(x,nb_filter):
      branch1x1 = Conv2d_BN(x,nb_filter,(1,1), padding='same',strides=(1,1),name=None)
  
      branch3x3 = Conv2d_BN(x,nb_filter,(1,1), padding='same',strides=(1,1),name=None)
      branch3x3 = Conv2d_BN(branch3x3,nb_filter,(3,3), padding='same',strides=(1,1),name=None)
  
      branch5x5 = Conv2d_BN(x,nb_filter,(1,1), padding='same',strides=(1,1),name=None)
      branch5x5 = Conv2d_BN(branch5x5,nb_filter,(1,1), padding='same',strides=(1,1),name=None)
  
      branchpool = MaxPooling2D(pool_size=(3,3),strides=(1,1),padding='same')(x)
      branchpool = Conv2d_BN(branchpool,nb_filter,(1,1),padding='same',strides=(1,1),name=None)
  
      x = concatenate([branch1x1,branch3x3,branch5x5,branchpool],axis=3)
  
      return x
  
  def GoogLeNet():
      inpt = Input(shape=(224,224,3))
      #padding = 'same'，填充为(步长-1）/2,还可以用ZeroPadding2D((3,3))
      x = Conv2d_BN(inpt,64,(7,7),strides=(2,2),padding='same')
      x = MaxPooling2D(pool_size=(3,3),strides=(2,2),padding='same')(x)
      x = Conv2d_BN(x,192,(3,3),strides=(1,1),padding='same')
      x = MaxPooling2D(pool_size=(3,3),strides=(2,2),padding='same')(x)
      x = Inception(x,64)#256
      x = Inception(x,120)#480
      x = MaxPooling2D(pool_size=(3,3),strides=(2,2),padding='same')(x)
      x = Inception(x,128)#512
      x = Inception(x,128)
      x = Inception(x,128)
      x = Inception(x,132)#528
      x = Inception(x,208)#832
      x = MaxPooling2D(pool_size=(3,3),strides=(2,2),padding='same')(x)
      x = Inception(x,208)
      x = Inception(x,256)#1024
      x = AveragePooling2D(pool_size=(7,7),strides=(7,7),padding='same')(x)
      x = Dropout(0.4)(x)
      x = Dense(1000,activation='relu')(x)
      x = Dense(1000,activation='softmax')(x)
      model = Model(inpt,x,name='inception')
      return model

• ResNet-50的Keras实现：

  def Conv2d_BN(x, nb_filter,kernel_size, strides=(1,1), padding='same',name=None):
      if name is not None:
          bn_name = name + '_bn'
          conv_name = name + '_conv'
      else:
          bn_name = None
          conv_name = None
  
      x = Conv2D(nb_filter,kernel_size,padding=padding,strides=strides,activation='relu',name=conv_name)(x)
      x = BatchNormalization(axis=3,name=bn_name)(x)
      return x
  
  def Conv_Block(inpt,nb_filter,kernel_size,strides=(1,1), with_conv_shortcut=False):
      x = Conv2d_BN(inpt,nb_filter=nb_filter[0],kernel_size=(1,1),strides=strides,padding='same')
      x = Conv2d_BN(x, nb_filter=nb_filter[1], kernel_size=(3,3), padding='same')
      x = Conv2d_BN(x, nb_filter=nb_filter[2], kernel_size=(1,1), padding='same')
      if with_conv_shortcut:
          shortcut = Conv2d_BN(inpt,nb_filter=nb_filter[2],strides=strides,kernel_size=kernel_size)
          x = add([x,shortcut])
          return x
      else:
          x = add([x,inpt])
          return x
  
  def ResNet50():
      inpt = Input(shape=(224,224,3))
      x = ZeroPadding2D((3,3))(inpt)
      x = Conv2d_BN(x,nb_filter=64,kernel_size=(7,7),strides=(2,2),padding='valid')
      x = MaxPooling2D(pool_size=(3,3),strides=(2,2),padding='same')(x)
      
      x = Conv_Block(x,nb_filter=[64,64,256],kernel_size=(3,3),strides=(1,1),with_conv_shortcut=True)
      x = Conv_Block(x,nb_filter=[64,64,256],kernel_size=(3,3))
      x = Conv_Block(x,nb_filter=[64,64,256],kernel_size=(3,3))
      
      x = Conv_Block(x,nb_filter=[128,128,512],kernel_size=(3,3),strides=(2,2),with_conv_shortcut=True)
      x = Conv_Block(x,nb_filter=[128,128,512],kernel_size=(3,3))
      x = Conv_Block(x,nb_filter=[128,128,512],kernel_size=(3,3))
      x = Conv_Block(x,nb_filter=[128,128,512],kernel_size=(3,3))
      
      x = Conv_Block(x,nb_filter=[256,256,1024],kernel_size=(3,3),strides=(2,2),with_conv_shortcut=True)
      x = Conv_Block(x,nb_filter=[256,256,1024],kernel_size=(3,3))
      x = Conv_Block(x,nb_filter=[256,256,1024],kernel_size=(3,3))
      x = Conv_Block(x,nb_filter=[256,256,1024],kernel_size=(3,3))
      x = Conv_Block(x,nb_filter=[256,256,1024],kernel_size=(3,3))
      x = Conv_Block(x,nb_filter=[256,256,1024],kernel_size=(3,3))
      
      x = Conv_Block(x,nb_filter=[512,512,2048],kernel_size=(3,3),strides=(2,2),with_conv_shortcut=True)
      x = Conv_Block(x,nb_filter=[512,512,2048],kernel_size=(3,3))
      x = Conv_Block(x,nb_filter=[512,512,2048],kernel_size=(3,3))
      x = AveragePooling2D(pool_size=(7,7))(x)
      x = Flatten()(x)
      x = Dense(1000,activation='softmax')(x)
      
      model = Model(inputs=inpt,outputs=x)
      return model

• DenseNet-121的Keras实现：

  def DenseNet121(nb_dense_block=4, growth_rate=32, nb_filter=64, reduction=0.0, dropout_rate=0.0, weight_decay=1e-4, classes=1000, weights_path=None):
      '''Instantiate the DenseNet 121 architecture,
          # Arguments
              nb_dense_block: number of dense blocks to add to end
              growth_rate: number of filters to add per dense block
              nb_filter: initial number of filters
              reduction: reduction factor of transition blocks.
              dropout_rate: dropout rate
              weight_decay: weight decay factor
              classes: optional number of classes to classify images
              weights_path: path to pre-trained weights
          # Returns
              A Keras model instance.
      '''
      eps = 1.1e-5
  
      # compute compression factor
      compression = 1.0 - reduction
  
      # Handle Dimension Ordering for different backends
      global concat_axis
      if K.image_dim_ordering() == 'tf':
        concat_axis = 3
        img_input = Input(shape=(224, 224, 3), name='data')
      else:
        concat_axis = 1
        img_input = Input(shape=(3, 224, 224), name='data')
  
      # From architecture for ImageNet (Table 1 in the paper)
      nb_filter = 64
      nb_layers = [6,12,24,16] # For DenseNet-121
  
      # Initial convolution
      x = ZeroPadding2D((3, 3), name='conv1_zeropadding')(img_input)
      x = Convolution2D(nb_filter, 7, 7, subsample=(2, 2), name='conv1', bias=False)(x)
      x = BatchNormalization(epsilon=eps, axis=concat_axis, name='conv1_bn')(x)
      x = Scale(axis=concat_axis, name='conv1_scale')(x)
      x = Activation('relu', name='relu1')(x)
      x = ZeroPadding2D((1, 1), name='pool1_zeropadding')(x)
      x = MaxPooling2D((3, 3), strides=(2, 2), name='pool1')(x)
  
      # Add dense blocks
      for block_idx in range(nb_dense_block - 1):
          stage = block_idx+2
          x, nb_filter = dense_block(x, stage, nb_layers[block_idx], nb_filter, growth_rate, dropout_rate=dropout_rate, weight_decay=weight_decay)
  
          # Add transition_block
          x = transition_block(x, stage, nb_filter, compression=compression, dropout_rate=dropout_rate, weight_decay=weight_decay)
          nb_filter = int(nb_filter * compression)
  
      final_stage = stage + 1
      x, nb_filter = dense_block(x, final_stage, nb_layers[-1], nb_filter, growth_rate, dropout_rate=dropout_rate, weight_decay=weight_decay)
  
      x = BatchNormalization(epsilon=eps, axis=concat_axis, name='conv'+str(final_stage)+'_blk_bn')(x)
      x = Scale(axis=concat_axis, name='conv'+str(final_stage)+'_blk_scale')(x)
      x = Activation('relu', name='relu'+str(final_stage)+'_blk')(x)
      x = GlobalAveragePooling2D(name='pool'+str(final_stage))(x)
  
      x = Dense(classes, name='fc6')(x)
      x = Activation('softmax', name='prob')(x)
  
      model = Model(img_input, x, name='densenet')
  
      if weights_path is not None:
        model.load_weights(weights_path)
  
      return model
  
  
  def conv_block(x, stage, branch, nb_filter, dropout_rate=None, weight_decay=1e-4):
      '''Apply BatchNorm, Relu, bottleneck 1x1 Conv2D, 3x3 Conv2D, and option dropout
          # Arguments
              x: input tensor 
              stage: index for dense block
              branch: layer index within each dense block
              nb_filter: number of filters
              dropout_rate: dropout rate
              weight_decay: weight decay factor
      '''
      eps = 1.1e-5
      conv_name_base = 'conv' + str(stage) + '_' + str(branch)
      relu_name_base = 'relu' + str(stage) + '_' + str(branch)
  
      # 1x1 Convolution (Bottleneck layer)
      inter_channel = nb_filter * 4  
      x = BatchNormalization(epsilon=eps, axis=concat_axis, name=conv_name_base+'_x1_bn')(x)
      x = Scale(axis=concat_axis, name=conv_name_base+'_x1_scale')(x)
      x = Activation('relu', name=relu_name_base+'_x1')(x)
      x = Convolution2D(inter_channel, 1, 1, name=conv_name_base+'_x1', bias=False)(x)
  
      if dropout_rate:
          x = Dropout(dropout_rate)(x)
  
      # 3x3 Convolution
      x = BatchNormalization(epsilon=eps, axis=concat_axis, name=conv_name_base+'_x2_bn')(x)
      x = Scale(axis=concat_axis, name=conv_name_base+'_x2_scale')(x)
      x = Activation('relu', name=relu_name_base+'_x2')(x)
      x = ZeroPadding2D((1, 1), name=conv_name_base+'_x2_zeropadding')(x)
      x = Convolution2D(nb_filter, 3, 3, name=conv_name_base+'_x2', bias=False)(x)
  
      if dropout_rate:
          x = Dropout(dropout_rate)(x)
  
      return x
  
  
  def transition_block(x, stage, nb_filter, compression=1.0, dropout_rate=None, weight_decay=1E-4):
      ''' Apply BatchNorm, 1x1 Convolution, averagePooling, optional compression, dropout 
          # Arguments
              x: input tensor
              stage: index for dense block
              nb_filter: number of filters
              compression: calculated as 1 - reduction. Reduces the number of feature maps in the transition block.
              dropout_rate: dropout rate
              weight_decay: weight decay factor
      '''
  
      eps = 1.1e-5
      conv_name_base = 'conv' + str(stage) + '_blk'
      relu_name_base = 'relu' + str(stage) + '_blk'
      pool_name_base = 'pool' + str(stage) 
  
      x = BatchNormalization(epsilon=eps, axis=concat_axis, name=conv_name_base+'_bn')(x)
      x = Scale(axis=concat_axis, name=conv_name_base+'_scale')(x)
      x = Activation('relu', name=relu_name_base)(x)
      x = Convolution2D(int(nb_filter * compression), 1, 1, name=conv_name_base, bias=False)(x)
  
      if dropout_rate:
          x = Dropout(dropout_rate)(x)
  
      x = AveragePooling2D((2, 2), strides=(2, 2), name=pool_name_base)(x)
  
      return x
  
  
  def dense_block(x, stage, nb_layers, nb_filter, growth_rate, dropout_rate=None, weight_decay=1e-4, grow_nb_filters=True):
      ''' Build a dense_block where the output of each conv_block is fed to subsequent ones
          # Arguments
              x: input tensor
              stage: index for dense block
              nb_layers: the number of layers of conv_block to append to the model.
              nb_filter: number of filters
              growth_rate: growth rate
              dropout_rate: dropout rate
              weight_decay: weight decay factor
              grow_nb_filters: flag to decide to allow number of filters to grow
      '''
  
      eps = 1.1e-5
      concat_feat = x
  
      for i in range(nb_layers):
          branch = i+1
          x = conv_block(concat_feat, stage, branch, growth_rate, dropout_rate, weight_decay)
          concat_feat = merge([concat_feat, x], mode='concat', concat_axis=concat_axis, name='concat_'+str(stage)+'_'+str(branch))
  
          if grow_nb_filters:
              nb_filter += growth_rate
  
      return concat_feat, nb_filter