1. Who is the “Father of Deep Learning”?
Dr. Charles C. Tappert
Seidenberg School of CSIS, Pace University

2. What is Deep Learning?
Fortune, 2016
Frank Rosenblatt

3. Why is Deep Learning Important?
Causing a revolution in Artificial Intelligence
Electrifying the computing industry
Transforming corporate America
Why? – because over the last five years we have experienced quantum leaps in the quality of many everyday technologies
Fortune, 2016
Frank Rosenblatt

4. Why is Deep Learning Important?
Major advances in Image Recognition
Search and automatically organize collections of photos
Apple, Amazon, Microsoft, Facebook
Speech Technologies work much better
Speech recognition: Apple’s Siri, Amazon’s Alexa, Microsoft’s Cortana, Chinese Baidu speech interfaces
Translation of spoken sentences: Google Translate
Deep learning also improving medical applications, robotics, autonomous drones, self-driving cars, etc.
Frank Rosenblatt

5. Major Types of Deep Learning Systems
Convolutional Neural Networks for matrix data
a type of feed-forward artificial neural network in which the connectivity pattern between its neurons is inspired by the organization of the animal visual cortex
often referred to as multilayer perceptrons
Recurrent Neural Networks for sequential data
a class of artificial neural network where connections between units form a directed cycle (feedback)
Frank Rosenblatt

6. Google Search on “Father of Deep Learning”
Results of Google search on “Father of Deep Learning”
Geoff Hinton for restricted Boltzmann machines stacked as deep-belief networks
Yann LeCun for convolutional neural networks
Yoshua Bengio, whose team is behind Theano (and whose research contributions are many)
Andrew Ng, who helped build Google Brain
Jürgen Schmidhuber, who developed recurrent nets and Long Short-Term Memory (LSTM), a recurrent neural network (RNN) architecture
Frank Rosenblatt for his creation of perceptrons
This presentation builds a case for naming Frank Rosenblatt as the “Father of Deep Learning”
Frank Rosenblatt

7. Dr. Frank Rosenblatt, 1928-1971 The “Father of Deep Learning”
PhD Experimental Psychology, Cornell, 1956
Developed neural networks called perceptrons
A probabilistic model for information
storage and organization in the brain
Key properties
Association or learning
Generalization to new patterns
Distributed memory
Biologically plausible brain model
Cornell Aeronautical Lab ( ), Cornell ( )
Frank Rosenblatt

8. Agenda Rosenblatt’s Nervous System Diagram to be Modeled
Perceptron Experimental Systems
The Mark I Perceptron – Visual System Model
The Tobermory Perceptron – Auditory System Model
Perceptron Computer Simulations
Rosenblatt's Book Principles of Neurodynamics
Rosenblatt-Minsky Debates and Minsky-Papert Book
Deep Learning Systems
Comparison of Deep Learning Systems with Perceptrons
Frank Rosenblatt

9. Nervous System (from Rosenblatt’s Principles of Neurodynamics)
Frank Rosenblatt

10. Definition and Classification of Perceptrons
A perceptron is a network of sensory (S), association (A), and response (R) units with an interaction matrix of connection coefficients for all pairs of units
A series-coupled perceptron is feed-forward S→A→R network
A cross-coupled perceptron is a system in which some connections join units in the same layer
A back-coupled perceptron is a system in which some connections flows back to an earlier layer
Frank Rosenblatt

11. Simple Perceptron Experimental System (from Rosenblatt’s Principles of Neurodynamics)
A simple perceptron is series-coupled with one R-unit and fixed S->A connections
Frank Rosenblatt

12. General Perceptron Experimental System (from Rosenblatt’s Principles of Neurodynamics)
A-unit network usually has several layers with possible cross-coupling
Frank Rosenblatt

13. The Mark I Perceptron Visual system model and pattern classifier
Examining A-unit of Mark I
(Rosenblatt on left)
Typical three-layer perceptron: fixed S→A and variable A→R connections
Frank Rosenblatt

14. The Mark I Perceptron Visual system model and pattern classifier
Sensory (input) layer of 400 photosensitive units in a 20x20 grid modeling a small retina
Connections from input to association layer altered through plug-board wiring, but once wired they were fixed for the duration of an experiment
Association (hidden) layer of 512 units (stepping motors) each of which could take several excitatory and inhibitory inputs
Connections from association to output layer were variable weights (motor-driven potentiometers) adjusted through error-propagating training process
Response (output) layer of 8 units
Frank Rosenblatt

15. The Mark I Perceptron Visual system model and pattern classifier
The Mark I Perceptron is now at the Smithsonian Institution
Frank Rosenblatt

16. The Tobermory Perceptron Auditory system model and pattern classifier
Named after talking cat,
Tobermory, in story by H.H. Munro (aka Saki)
Large machine at that time
S-units: 45 band-pass filters and 80 difference detectors
A-units: 1600 A1-units (20 time samples per detector) & 1000 A2-units
R-units: 12, with 12,000 adaptive weights A2→R-units.
Frank Rosenblatt

17. Perceptron Computer Simulations
Hardware implementations made good demonstrations but software simulations were far more flexible
In the 1960s these computer simulations required machine language coding for speed and memory usage
Simulation software package – user could specify the number of layers, the number of units per layer, type of connections between layers, etc.
Computer time at Cornell and NYU
Frank Rosenblatt

18. Rosenblatt's Book Principles of Neurodynamics, 1962
Part I: historical review of brain modeling approaches, physiological and psychological considerations, and basic definitions and concepts of the perceptron approach
Part II: three-layer, series-coupled perceptrons –mathematical underpinnings and experimental results
Part III: multi-layer and cross-coupled perceptrons
Part VI: back-coupled perceptrons
Book used to teach an interdisciplinary course "Theory of Brain Mechanisms" that drew students from Cornell's Engineering and Liberal Arts colleges
Frank Rosenblatt

19. Series-Coupled Perceptrons
A perceptron is a network of sensory (S), association (A), and response (R) units with an interaction matrix of connection coefficients for all pairs of units
A series-coupled perceptron is feed-forward S→A→R network
An simple perceptron is series-coupled with one R-unit connected to every A-unit and fixed S→A connections
Frank Rosenblatt

20. Series-Coupled Perceptron Theorems
Convergence Theorem: Given a simple perceptron, a stimulus world W, and any classification C(W) for which a solution exists, then if all stimuli in W re-occur in finite time, the error correction procedure will always find a solution
Perceptrons were the first neural networks that could learn the weights!
Solution Existence Theorem: The class of simple perceptrons for which a solution exists to every classification C(W) of possible environments W is non-empty
There exists an S→A→R feedforward perceptron that can solve any classification problem!
Frank Rosenblatt

21. Series-Coupled Perceptrons Mark I was typical S→A→R perceptron
Connections
S→A: fixed, usually local
A→R: adjustable w training
Frank Rosenblatt

22. Series-Coupled Perceptrons (from Rosenblatt’s Principles of Neurodynamics)
A1 biologically-plausible detector units: broken lines indicate inhibitory fields, solid lines excitatory fields
Frank Rosenblatt

23. Series-Coupled Perceptrons (from Rosenblatt’s Principles of Neurodynamics)
A2 units often combinations of A1 units
Frank Rosenblatt

24. Series-Coupled Perceptrons (from Rosenblatt’s Principles of Neurodynamics)
Rosenblatt studied three and four-layer series-coupled perceptrons with two sets of variable weights but was unable to find a suitable training procedure like back-propagation
Dotted lines are variable connections
Frank Rosenblatt

25. Cross-Coupled Perceptrons (from Rosenblatt’s Principles of Neurodynamics)
A cross-coupled perceptron is a system in which some connections join units of the same type (S, A, and/or R)
Frank Rosenblatt

26. Back-Coupled Perceptrons (from Rosenblatt’s Principles of Neurodynamics)
A back-coupled perceptron is a system with feedback paths from units located near the output end of the system to units closer to the sensory end
Frank Rosenblatt

27. Rosenblatt-Minsky Debates and Minsky-Papert Book
Rosenblatt and Marvin Minsky (MIT) debated at conferences the value of biologically inspired computation, Rosenblatt arguing that his neural networks could do almost anything and Minsky countering that they could do little
Minsky, wanting to decide the matter once and for all, collaborated with Seymour Papert and published a book in 1969, Perceptrons: An Introduction to Computational Geometry, where they asserted about perceptrons (page 4), "Most of this writing ... is without scientific value...”
Minsky, although well aware that powerful perceptrons have multiple layers and Rosenblatt's basic feed-forward perceptrons have three layers, defined a perceptron as a two-layer machine that can handle only linearly separable problems and, for example, cannot solve the exclusive-OR problem
Precipitated an Artificial Intelligence Winter that lasted about 15 years
Frank Rosenblatt

28. Three Wave Development of Deep Learning (Deep Learning by Goodfellow, Bengio, and Courville)
1940s-1960s: Early Neural Networks (Cybernetics?)
Rosenblatt’s perceptron – developed from Hebb’s synaptic strengthening ideas and McCulloch-Pitts Neuron
Key idea – variations of stochastic gradient descent
Wave killed by Minsky 1969, lead to “AI Winter”
1980s-1990s: Connectionism
Rumelhart, et al.
Key idea – backpropagation
2006-present: Deep Learning
Started with Hinton’s deep belief network
Key idea – hierarchy of many layers in the neural network

29. Deep Learning Real-World Impact (Deep Learning by Goodfellow, Bengio, and Courville)
A dramatic moment in the meteoric rise of deep learning came when a convolutional network won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) for the first time – Krizhevsky, et al., 2012
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) now consistently won by deep networks

30. Deep Learning Systems Convolutional Neural Networks (CNN) (Deep Learning by Goodfellow, Bengio, and Courville)
The award-winning, deep learning systems are multilayer feed-forward, convolutional networks (CNNs)
Edge detection example – sparse fixed shared weights
Frank Rosenblatt

31. Deep Learning Systems Recurrent Neural Networks (RNN) (Deep Learning by Goodfellow, Bengio, and Courville)
Recurrent Neural Networks: connections within the same layer and/or back to earlier layers – used for sequence recognition
Frank Rosenblatt

32. Deep Learning CNNs vs Perceptrons Award-winning CNNs
The award-winning, deep learning systems are multilayer feed-forward, convolutional neural networks (CNNs)
These are just multilayer perceptrons with fixed, biologically-inspired connections in the early layers
And it is interesting that they are generally called multilayer perceptrons (MLP)
Frank Rosenblatt

33. Deep Learning RNNs vs Perceptrons Recurrent Neural Networks
Recurrent Neural Networks have connections within the same layer and/or connections back to earlier layers
These are just cross-coupled and back-coupled perceptrons
Frank Rosenblatt

34. Deep Learning Systems vs Perceptrons Conclusions
Although conceptually the same as perceptrons, the deep learning networks have the advantages of
Increased computing power + GPUs (50 years from 1960s)
Massive training data (big data) not available in 1960s
Backpropagation algorithm not available in 1960s
Allows training of complete system, pulling everything together
Nevertheless, we make a strong case for recognizing Frank Rosenblatt as the “Father of Deep Learning”
Frank Rosenblatt

35. References
Roger Parloff, Why Deep Learning is Suddenly Changing Your Life, Fortune, 2016
Frank Rosenblatt, Principles of Neurodynamics, Spartan, 1962
Marvin Minsky and Seymour Papert, Perceptrons: An Introduction to Computational Geometry, MIT Press, 1969
Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, MIT Press, 2016
Various links to Wikipedia
Frank Rosenblatt