Goodfellow: Chap 1 Introduction Dr. Charles Tappert The information here, although greatly condensed, comes almost entirely from the chapter content.

Artificial Intelligence Early Days – AI solved problems easy for computers but difficult for humans Problems described by formal, math rules The AI challenge was to solve problems easy for humans but difficult to describe formally Recognizing spoken words or faces in images Deep learning is abut solving these more intuitive problems Allowing computers to learn from experience Building a hierarchy of concepts, each defined by simpler concepts

Machine Learning Early AI successes were in formal environments IBM’s Deep Blue beats world champion Kasparov Knowledge-based AI attempted to hard-code world knowledge in formal languages Difficulties faced suggested need for ability to acquire knowledge by extracting patterns from real-world raw data This capability is known as machine learning Examples are logistic regression and naïve Bayes

Data Representation Performance of machine learning algorithms depends heavily on the data representation The features/attributes characterizing the data This dependence on representations is a general phenomenon in computer science and even in daily life Many AI tasks can be solved by designating the right set of features

Representation Learning One solution to the data representation problem is to have machine learning discover the representation Example: Autoencoder – combination of encoder to convert the input data into another representation, and decoder to convert back to the original format Goal is usually to separate the factors of variation that explain the observed data To disentangle and discard those not of interest

Deep Learning Deep learning solves the representation problem by introducing representations expressed in terms of simpler representations Deep learning involves a hierarchy of concepts that allows the computer to learn complicated concepts by building them out of simpler ones Graphically the concepts are built on top of each other with many layers The quintessential example of a deep learning model is the feedforward deep network or MLP

Illustration of Deep Learning Model

Deep Learning Deep learning is a machine learning approach to AI that allows computers to improve with experience and data Deep learning achieves great power and flexibility by learning to represent the world as a nested hierarchy of concepts, with each concept defined in relation to simpler ones The relationship among these different AI disciplines is shown in the following figure

Venn Diagram: Deep Learning

Flow Chart: Deep Learning

Goodfellow Textbook Part I Part II Part III Basic math tools and machine learning concepts Part II Established deep learning algorithms Part III More speculative ideas for future research

Goodfellow Textbook Organization

Three Waves of Development of Deep Learning 1940s-1960s: Early Neural Networks (Cybernetics?) Rosenblatt’s perceptron – developed from Hebb’s synaptic strengthening ideas and McCulloch-Pitts Neuron Key idea – variations of stochastic gradient descent Wave killed by Minsky 1969, lead to “AI Winter” 1980s-1990s: Connectionism Rumelhart, et al. Key idea – backpropagation 2006-present: Deep Learning Started with Hinton’s deep belief network Key idea – hierarchy of many layers in the neural network

First Two Waves

Deep Learning Increasing Dataset Sizes Since the 1990s machine learning systems have been used successfully in commercial applications But regarded as being more of an art than a technology Deep learning is regarded more and more as a technology – the amount of development skill reduces as the amount of training data increases The age of “Big Data” has made data collection easier As of 2016, rule of thumb is that supervised deep learning algorithms need around 5k labeled samples per category Performance is expected to match or exceed the human with a dataset of at least 10 million labeled samples

Deep Learning Increasing Dataset Sizes

Deep Learning Increasing Dataset Sizes Example inputs from MNIST dataset

Deep Learning Increasing Model Sizes A key reason deep learning networks are wildly successful today is that greater computation resources have allowed rapidly increasing model sizes Biological neurons are not especially densely connected and the number of connections per neuron in machine learning models have been within an order of magnitude of mammalian brains for decades As for the increase in the total number of model neurons, which doubles roughly every 2.4 years, we have decades to go at this rate before reaching the number on neurons in the human brain

Deep Learning Increasing Model Sizes: Connections per Neuron 9 = Commodity Off-The-Shelf High Performance Computing (COTS HPC) technology, 2013

Deep Learning Increasing Model Sizes: Number of Neurons 19 = Commodity Off-The-Shelf High Performance Computing (COTS HPC) technology, 2013

Deep Learning Increasing Accuracy, Complexity and Real-World Impact Since the 1980s machine learning systems have consistently improved recognition accuracy And applied with success to broader sets of applications In terms of complexity Early models recognized individual objects in small images Today, we process large high-resolution photos and typically recognize 1000 different categories of objects A dramatic moment in the meteoric rise of deep learning came when a convolutional network won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) for the first time – Krizhevsky, et al., 2012

Deep Learning Increasing Accuracy, Complexity and Real-World Impact ImageNet Large Scale Visual Recognition Challenge (ILSVRC) now consistently won by deep networks