LeNet-5：Gradient-Based Learning Applied to Document Recognition

1998年的LeNet-5是CNN的经典之作，但是该模型在后来未能火起来，主要原因是当时的计算力不足和数据量的不足。并且当时的SVM在可承受计算量的情况下，达到，甚至超过了神经模型。CNN虽然在当时没能受到重视，但是并没有掩盖CNN的强大能力。

下图是LeNet的架构图：

LeNet-5架构图

LeNet由两层conv，两层pool，三层fc组成。

以下是用TensorFlow和Keras混合编写的LeNet-5模型：

import tensorflow as tf
keras = tf.keras
from tensorflow.python.keras.layers import Conv2D,MaxPool2D,Dropout,Flatten,Dense

def inference(inputs,
              num_classes=10,
              is_training=True,
              dropout_keep_prob=0.5):
  '''

  inputs: a tensor of images
  num_classes: the num of category.
  is_training: set ture when it used for training
  dropout_keep_prob: the rate of dropout during training
  '''

  x = inputs
  # conv1
  x = Conv2D(6, [5,5], 1, activation='relu', name='conv1')(x) 
  # pool1
  x = MaxPool2D([2,2], 2, name='pool1')(x)
  # conv2
  x = Conv2D(16, [5,5], 1, activation='relu', name='conv2')(x)
  # pool2
  x = MaxPool2D([2,2], 2, name='pool2')(x)
  x = Flatten(name='pool2_flatten')(x)
  if is_training:
    x = Dropout(rate=dropout_keep_prob)(x)
  # fc3
  x = Dense(120, activation='relu', name='fc3')(x)
  if is_training:
    x = Dropout(rate=dropout_keep_prob)(x)
  # fc4
  x = Dense(84, activation='relu', name='fc4')(x)
  # logits
  logits = Dense(num_classes, activation='softmax')(x)
  return logits

if __name__ == '__main__':
  x = tf.placeholder(tf.float32, [None, 784])
  images = tf.reshape(x,[-1,28,28,1])
  labels = tf.placeholder(tf.float32, [None, 10])
  dropout_keep_prob = tf.placeholder(tf.float32)
  logits = inference(inputs=images,
                    num_classes=10,
                    is_training=True,
                    dropout_keep_prob=dropout_keep_prob)

  with tf.variable_scope('costs'):
    cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(
        logits=logits, labels=labels), name='xent')

  with tf.variable_scope('train'):
    train_op = tf.train.AdamOptimizer().minimize(cost)

  from tensorflow.examples.tutorials.mnist import input_data
  mnist_data = input_data.read_data_sets('./mnist_data', one_hot=True)

  acc_test = tf.divide(
      tf.reduce_sum(keras.metrics.categorical_accuracy(labels,logits)),
      len(mnist_data.test.labels))

  with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    Writer = tf.summary.FileWriter('./tmp',sess.graph)

    for i in range(200):
      batch_x,batch_y = mnist_data.train.next_batch(50)
      _, c = sess.run([train_op, cost], feed_dict={
          x:batch_x, labels:batch_y, dropout_keep_prob:0.5})
      acc_test1 = sess.run(acc_test, feed_dict={
          x:mnist_data.test.images, labels:mnist_data.test.labels, dropout_keep_prob:1})
      print('step:%04d  cost:%.4f  test_acc:%.3f'%(i+1,c,acc_test1))

代码以Keras来实现模型的inference过程，其他部分使用Tensorflow，这样可以大大减少构建模型的复杂度。
LeNet-5的具体配置：