Key points form this chapter

• Neocognitron(Fukushima, 1980), introduced a powerful model architecture for processing images, which is inspired by the structure of the mammalian visual system and later become the basis for the modern convolutional network.
• Geoffrey Hinton showed that deep belief network could be efficiently trained using strategy called greedy layer-wise pretraining, and other research groups find out, the same strategy can be used to train many other kinds of network
• supervised deep learning algorithm will generally achieve acceptable performance with around 5000 labeled examples per category, can exceed human performance with over 10 million labeled examples.
• Goodfellow et al. (2014d) showd that neural networks could learn to generate an entire sequence of characters.
• RNN, such as LSTM, can be used to learn relationships between sequences and other sequences, and has many applications to machine translation.
• One important concept mentioned in this chapter is neural turing machine(Graves et al.,2014a), that learn to read from memory cells and write arbitrary content to memory cells. It can learn to program, such as sorting a sequence of numbers.

My comments

People understand the word by features. When we receive inputs from the world, we not really obtain the full information from the input, instead, we extract feature from the input. The input from the world is discrete, but we feel that the world the continuous, that's because we only learn features from the input. Autoencoder might be a candidate for extracting feature from input, it can rebuild input from the low dimensional representation, just like we can remodel the world from our understanding of the world. I should keep thinking on this subject. Find a elementary of features that is essential to understand the world.