1. Prof. János LEVENDOVSZKY (levendov@hit.bme.hu)
NEURAL NETWORKS
Lecturer:
Prof. János LEVENDOVSZKY

2. Course information
One major test scheduled to the end of the semester;
Several class-room quizzes;
Exam;
Grade=0.5*course performance+0.5* exam

3. Suggested literature and references
Haykin, S.: Neural networks a comprehensive foundation, MacMillan, 2004
Hassoun, M.: Fundamentals of artificial neural networks, MIT Press, 1995
Chua, L.O., Roska T. and Venetianer, P.L.: "The CNN is as Universal as theTuring Machine", IEEE Trans. on Circuits and Systems, Vol. 40., March, 1993
J.G. Proakis: Digital communications,McGraw Hill, 1996
Lecture notes (most important)!!!

4. Scope & motivations Representation ? Learnig ? Generalization ?
OPTIMIZATION ???
Representation ? Learnig ? Generalization ?
Solution
Modeling architecture (signal processing elements with free parameters)
estimated output
Untractable by analytical means (huge amount of free parameters & data to be taken into account)
Some input data
desired output
Highly complex systems in Information Technologies + -
error signal
KNOWLEDGE TRANSFER
Optimal operation
Modeling architecture with optimized parameters
new input
general solution

5. Complex task requiring high level modeling
Neural approach
state of the game
next move
Complex task requiring high level modeling
FFNN
Input data
output

6. HOW TO MODEL AND OPTIMIZE THESE SYSTEMS ?
Examples
Endeavour:
HOW TO MODEL AND OPTIMIZE THESE SYSTEMS ?
IP network (with routers, switches, buffers …etc.)
Input traffic volume (voice, video, multimedia)
QoS parameters (average packet loss rate, average packet delay)
GSM or UMTS, b3G systems
Number of users
Multiuser interference
Financial system (stock market, economical factors)
Stock prices, currency exchange rates (financial data series )
Optimal investment for maximizing the return

7. A simple example – packet delay estimation
Interarrival time
Complex system (Packet switched network)
Input traffic
delay
Modeling architecture y=Ax+B
est. delay
Measurements d
Future x
est. delay
learning
generalization

8. A simple example – packet delay estimation (cont’)
Complex system (Packet switched network)
Input traffic
delay
Modeling architecture y=Ax^3+bx^2+Cx+D
est. delay
Measurements d
Linear appr. is no good

9. Problems
Linear modeling is very poor (most of the real-life problems are of highly nonlinear nature), what about more complex representation ?
How to develop fast learning algorithms ?
How to develop exact measures expressing the quality of generalization ?

10. Solution provided by evolution and biology: MAMMAL BRAIN
The challenge
Modeling architecture
Representation
Learning
Generalization
Robustness
Modularity
Solution provided by evolution and biology: MAMMAL BRAIN
-high representation capability;
large scale adaptation;
far reaching generalizations;
modular structure (nerve cells, neurons);
very robust

11. Copying the brain ?
Neurobio-logical models
Feature extraction
Artificial neural networks
far too complex
gross simplifications only preserving the computational aspects
Biological (brain):
Surface: 2500 cm2
Squishy
Neurons: 20 billion
Synapses: 240 trillion
Neuron size: 15 um
Synapse size: 1um
Synaptic ops: 30 trillion
NN platforms (computers):
Surface: 90 mm2
Crystalline
Transistors: 291 million
Transistor size 65nm
FLOPS: 25 billion
Computational model of the brain = Artificial Neural Networks (algorithms operating on the basis of some simplified computational principles of the brain) !
Brains still far outperform ANNs. As a result, our goal cannot be so ambitious as to construct general AI but to design algorithms which can perform better than brain in certain specific areas !

12. Neural network - interpretation
Collection of algorithms to solve highly complex problems in real-time (in the field of IT) by using novel computational paradigms routed in biology.

13. Historical notes
artifical neuron model, 40’s (McCulloch-Pitts, J. von Neumann);
perceptron learning rule, 50’s (Rosenblatt);
ADALINE, 60’s (Widrow)
critical review ,70’s (Minsky)
feedforwrad neural nets, 80’s (Cybenko, Hornik, Stinchcombe..)
back propagation learning, 80’s (Sejnowsky, Grossberg)
Hopfield net, 80’s (Hopfield, Grossberg);
self organizing feature map, 70’s - 80’s (Kohonen)
CNN, 80’s-90’s (Roska, Chua)
PCA networks, 90’s (Oja)
applictions in IT, 90’s - 00’s