Overview of deep learning Gregery T. Buzzard

Varieties of AI/ML AI is most general Many types of ML algs: Nearest Neighbor Naive Bayes Decision Trees Linear Regression Support Vector Machines (SVM) Neural Networks Deep learning is associated with neural networks

Machine learning Machine learning changes the paradigm from programming for answers to programming to discover rules (how to transform new data into new answers).

Shallow vs deep Shallow learning – only 1 or 2 layers: limited ability to combine features into more sophisticated features. E.g., the concept of an eye. Deep learning allows a model to learn all layers of a representation jointly rather than sequentially - better than stacking shallow models. Two key points: Layer-by-layer way in which increasingly complex representations are developed These representations are learned jointly

NN depth Deep learning uses multiple layers of processing Promotes reuse of rules for multiple inputs – i.e., identify common features rather than global properties. Promotes increasing levels of abstraction with depth.

NN depth Example activations in simple NN Layers farther from input are more abstract

NN overview Simple layers use a linear transformation (AX + b) plus an activation function (relu). The weights (parameters) A and b determine behavior – need to learn them.

NN overview Compare predictions with training targets using a loss function (or error function). Often CE or MSE

NN overview Use the loss function with an optimizer to update the weights. E.g., gradient descent.

Driving forces Three technical forces are driving advances in machine learning: Hardware Datasets and benchmarks Algorithmic advances (and software platforms) Belief that it works

Neural networks – hype is not new! Developed by Frank Rosenblatt in 1957 at Cornell under a grant from the Office of Naval Research. The New York Times reported the perceptron to be "the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.” https://en.wikipedia.org/wiki/Perceptron https://www.dexlabanalytics.com/wp-content/uploads/2017/07/ The-Timeline-of-Artificial-Intelligence-and-Robotics.png

Neural networks - roadblocks In 1969 Minsky and Papert of MIT claimed that Rosenblatt’s predictions had been grossly exaggerated.  Several booms and busts later, lots of data, faster computers, and good programming infrastructure led to many of the early predictions coming true. What next? https://qph.fs.quoracdn.net/main-qimg-6c76e47604e22987a0349e2d4c892646

Example code on Google colaboratory