1. Goodfellow: Chap 1 Introduction
Dr. Charles Tappert
The information here, although greatly condensed, comes almost entirely from the chapter content.

2. Artificial Intelligence
Early Days – AI solved problems easy for computers but difficult for humans
Problems described by formal, math rules
The AI challenge was to solve problems easy for humans but difficult to describe formally
Recognizing spoken words or faces in images
Deep learning is abut solving these more intuitive problems
Allowing computers to learn from experience
Building a hierarchy of concepts, each defined by simpler concepts

3. Machine Learning Early AI successes were in formal environments
IBM’s Deep Blue beats world champion Kasparov
Knowledge-based AI attempted to hard-code world knowledge in formal languages
Difficulties faced suggested need for ability to acquire knowledge by extracting patterns from real-world raw data
This capability is known as machine learning
Examples are logistic regression and naïve Bayes

4. Data Representation
Performance of machine learning algorithms depends heavily on the data representation
The features/attributes characterizing the data
This dependence on representations is a general phenomenon in computer science and even in daily life
Many AI tasks can be solved by designating the right set of features

5. Representation Learning
One solution to the data representation problem is to have machine learning discover the representation
Example: Autoencoder – combination of encoder to convert the input data into another representation, and decoder to convert back to the original format
Goal is usually to separate the factors of variation that explain the observed data
To disentangle and discard those not of interest

6. Deep Learning
Deep learning solves the representation problem by introducing representations expressed in terms of simpler representations
Deep learning involves a hierarchy of concepts that allows the computer to learn complicated concepts by building them out of simpler ones
Graphically the concepts are built on top of each other with many layers
The quintessential example of a deep learning model is the feedforward deep network or MLP

7. Illustration of Deep Learning Model

8. Deep Learning
Deep learning is a machine learning approach to AI that allows computers to improve with experience and data
Deep learning achieves great power and flexibility by learning to represent the world as a nested hierarchy of concepts, with each concept defined in relation to simpler ones
The relationship among these different AI disciplines is shown in the following figure

9. Venn Diagram: Deep Learning

10. Flow Chart: Deep Learning

11. Goodfellow Textbook Part I Part II Part III
Basic math tools and machine learning concepts
Part II
Established deep learning algorithms
Part III
More speculative ideas for future research

12. Goodfellow Textbook Organization

13. Three Waves of Development of Deep Learning
1940s-1960s: Early Neural Networks (Cybernetics?)
Rosenblatt’s perceptron – developed from Hebb’s synaptic strengthening ideas and McCulloch-Pitts Neuron
Key idea – variations of stochastic gradient descent
Wave killed by Minsky 1969, lead to “AI Winter”
1980s-1990s: Connectionism
Rumelhart, et al.
Key idea – backpropagation
2006-present: Deep Learning
Started with Hinton’s deep belief network
Key idea – hierarchy of many layers in the neural network

14. First Two Waves

15. Deep Learning Increasing Dataset Sizes
Since the 1990s machine learning systems have been used successfully in commercial applications
But regarded as being more of an art than a technology
Deep learning is regarded more and more as a technology – the amount of development skill reduces as the amount of training data increases
The age of “Big Data” has made data collection easier
As of 2016, rule of thumb is that supervised deep learning algorithms need around 5k labeled samples per category
Performance is expected to match or exceed the human with a dataset of at least 10 million labeled samples

16. Deep Learning Increasing Dataset Sizes

17. Deep Learning Increasing Dataset Sizes Example inputs from MNIST dataset

18. Deep Learning Increasing Model Sizes
A key reason deep learning networks are wildly successful today is that greater computation resources have allowed rapidly increasing model sizes
Biological neurons are not especially densely connected and the number of connections per neuron in machine learning models have been within an order of magnitude of mammalian brains for decades
As for the increase in the total number of model neurons, which doubles roughly every 2.4 years, we have decades to go at this rate before reaching the number on neurons in the human brain

19. Deep Learning Increasing Model Sizes: Connections per Neuron
9 = Commodity Off-The-Shelf High Performance Computing (COTS HPC) technology, 2013

20. Deep Learning Increasing Model Sizes: Number of Neurons
19 = Commodity Off-The-Shelf High Performance Computing (COTS HPC) technology, 2013

21. Deep Learning Increasing Accuracy, Complexity and Real-World Impact
Since the 1980s machine learning systems have consistently improved recognition accuracy
And applied with success to broader sets of applications
In terms of complexity
Early models recognized individual objects in small images
Today, we process large high-resolution photos and typically recognize 1000 different categories of objects
A dramatic moment in the meteoric rise of deep learning came when a convolutional network won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) for the first time – Krizhevsky, et al., 2012

22. Deep Learning Increasing Accuracy, Complexity and Real-World Impact
ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
now consistently won by deep networks