1. Joost N. Kok Universiteit Leiden
Neural Networks
Joost N. Kok
Universiteit Leiden

2. Neural Networks
Book: Introduction to the theory of Neural Computation by Hertz, Krogh, Palmer
Website:
Additional Material: Powerpoint Sheets, Journal Articles, practical exercises
Other recommended books: Haykin, Bishop

3. The Von Neumann architecture

4. The Hungarian-born mathematician, John von Neumann (1903-1957)

5. The biological architecture

6. Biological computers Five distinguishing properties: Highly parallel
Robust and fault tolerant
Adaptive
Deals with fuzzy, noisy information
Small, compact

7. Graceful Degradation
performance
damage

8. Neurons

9. Brain consists of 100000000000 (1011) neurons

10. Neural activity
out in

11. Artificial Neuron

12. Input-output function
f(x) =
1 + e -x/a 1
nonlinear function:
a  0
f(e)
a   e

13. Artificial Connections (Synapses)
wAB
The weight of the connection from neuron A to neuron B A B
wAB

14. Supervised Networks

15. Example

16. Real Neurons Nonlinear Summation Sequences of pulses
No fixed time-delay

17. History 1943: McCulloch and Pitts: artificial neuron
1960: Rosenblatt: perceptrons
1969: Minsky and Papert: XOR problem
197?: Associative content-addressable memory
198?: Hopfield Networks, Boltzmann Networks
1985: Backpropagation learning rule
2000+ : Spiking networks, Support Vector Machines

18. Hopfield
McCulloch
Pitts
Papert
Minsky
Boltzmann
Hebb

20. Issues Neurocomputing vs. Neuroscience Types of Learning: Supervised
Unsupervised
Reinforcement

21. Research Questions Design: what is best architecture?
Learning: find good algorithms
Analysis: what is the power of networks?
Implementation: how should the network be implemented?