CV Recognition: From AlexNet To Inception-v4

姚伟峰

做研究就像比武论剑一样，要论剑就要到华山论剑，如果你一定要去太行山论剑，去挺进大别山，那别人只能当你是游击队，永远也别想成正规军。在计算机视觉领域，农村是永远也包围不了城市的。华山以外，很难论出好剑。
　　 　　 　　　　　　　　　　　　　　　　　　　 　- 汤晓鸥

AlexNet

Year

• 2012

Achievement

• ILSVRC-2012 winner, achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry in ILSVRC-2012 competition.

Current Affiliation

• Toronto University Google
  
  
  (right: Hinton, mid: Alex, left: Ilya Sutskever)

Features

• Bring deep learning back to CV community & industry.

Topology

GoogLeNet-v1

Year

• 2015 CVPR

Achievement

• ILSVRC-2014 winner with top-5 test error rate of 7.9%

Current Affiliation

• Google (Christian Szegedy)

Features

• More Accurate(Representative)

  • Wider

    • Introduce Inception-v1 (Deep Dream) with heterogeneous combination of convolutions

      
      

  • Deeper

    • 22 layers while AlexNet is 8
• Faster
  • Special designed Inception to decrease computation
  • Less parameters 4M while AlexNet is 61M (only 1 FC layer)

VGG

Year

• 2015 ICLR

Achievement

• ILSVRC-2014 runner-up with top-5 test error rate of 7.3%

Current Affiliation

• Oxford University Google (Karen Simonyan)

Features

• More Accurate(Representative)
  • Wider
    • feature map number up to 512
  • Deeper
    • 16(VGG-16) and 19(VGG-19)
• Faster
  • Simple factorization: use multiple 3x3 kernel to simulate bigger kernel. (2 to simulate 5x5, 3 to simulate 7x7)
  • No LRN is involved

While, VGG greatly increased the parameter number, from 61M(AlexNet) to 138M(VGG-16) and 144M(VGG-19).

Inception-v2 & Inception-v3

Year

• 2015 Dec

Achievement

• top-5 test error rate of 5.6% (v3)

Current Affiliation

• Google (Christian Szegedy)

Features

• More Accurate(Representative)
  • Wider
    • New inception modules
  • Deeper
    • v3 depth 17 if treating Inception as one, 47 layers in fact.
• More Accurate through tricks
  • Batch Normalization - v2, v3

    • location

    • algorithm

  • Label Smooth - v3
  • BN auxiliary classifier - v3
• Faster

  • Factorization: - v3

  • Grid Size Reduction - v3

    
    

  • Batch Normalization - v2, v3

Arch

• Inception-v2
  • v1 with BN layers

• Inception-v3

ResNet

Year

• 2015 Dec

Achievement

• ILSVRC-2015 winner with top-5 test error rate of 5.7%

Current Affiliation

• Microsoft Facebook (He Kaiming)

Features
Try to fix the bad behavior of CNN in linear component representation.

• Shortcut
  • CNN to approximate non-linear part while shortcut to simulate linear part
• More Accurate(representative)
  • Wider
    • feature map number up to 3072
  • Deeper
    • up to 152 layer
• Faster
  • Small kernel: all 3x3 except first layer(7x7)
  • Only one FC layer with 100M parameters in 152-layer arch

Arch

Inception-v4

Year

• 2016

Achievement

• top-5 test error rate of 4.2%

Current Affiliation

• Google (Christian Szegedy)

Arch