1. Neural Network Architectures Aydın Ulaş 02 December 2004 ulasmehm@boun.edu.tr

2. Outline Of Presentation Introduction Neural Networks Neural Network Architectures Conclusions

3. Introduction Some numbers… –The human brain contains about 10 billion nerve cells (neurons) –Each neuron is connected to the others through 10000 synapses Brain as a computational unit –It can learn, reorganize from experience –It adapts to the environment –It is robust and fault tolerant –Fast computations with too much individual computational units

4. Introduction Taking the nature as a model. Consider the neuron as a PE A neuron has –Input (dendrites) –Output (the axon) The information circulates from the dendrites to the axon via the cell body Axon connects to dendrites via synapses –Strength of synapses change –Synapses may be excitatory or inhibitory

5. Perceptron (Artificial Neuron) Definition : Non linear, parameterized function with restricted output range

6. Activation Functions Linear Sigmoid Hyperbolic tangent

7. Neural Networks A mathematical model to solve engineering problems –Group of highly connected neurons to realize compositions of non linear functions Tasks –Classification –Clustering –Regression According to input flow –Feed forward Neural Networks –Recurrent Neural Networks

8. Feed Forward Neural Networks The information is propagated from the inputs to the outputs Time has no role (Acyclic, no feedbacks from outputs to inputs)

9. Recurrent Networks Arbitrary topologies Can model systems with internal states (dynamic ones) Delays can be modeled More difficult to train Problematic performance –Stable Outputs may be more difficult to evaluate –Unexpected behavior (oscillation, chaos, …) x1x2

10. Learning The procedure that consists in estimating the parameters of neurons (setting up the weights) so that the whole network can perform a specific task. 2 types of learning –Supervised learning –Unsupervised learning The Learning process (supervised) –Present the network a number of inputs and their corresponding outputs (Training) –See how closely the actual outputs match the desired ones –Modify the parameters to better approximate the desired outputs –Several passes over the data

11. Supervised Learning The real outputs of the model for the given inputs is known in advance. The networks task is to approximate those outputs. A “Supervisor” provides examples and teach the neural network how to fulfill a certain task

12. Unsupervised learning Group typical input data according to some function. Data clustering No need of a supervisor – Network itself finds the correlations between the data –Examples: Kohonen feature maps (SOM)

13. Properties of Neural Networks Supervised networks are universal approximators (Non recurrent networks) Can act as –Linear Approximator (Linear Perceptron) –Nonlinear Approximator (Multi Layer Perceptron)

14. Other Properties Adaptivity –Adapt weights to the environment easily Ability to generalize –May provide against lack of data Fault tolerance –Not too much degradation of performances if damaged  The information is distributed within the entire net.

15. An Example Regression

16. Example Classification Handwritten digit recognition 16x16 bitmap representation –Converted to 1x256 bit vector 7500 points on training set 3500 points on test set 0000000001100000 0000000110100000 0000000100000000 0000001000000000 0000010000000000 0000100000000000 0001000111110000 0001011000011000 0001100000001000 0001000000001000 0000100000010000 0000011111110000

17. Training Try to minimize an error or cost function Backpropogation algorithm –Gradient Descent Learn the weights of the network Update the weights according to the error function

18. Applications Handwritten Digit Recognition Face recognition Time series prediction Process identification Process control Optical character recognition Etc…

19. Neural Networks Neural networks are statistical tools –Adjust non linear functions to accomplish a task –Need of multiple and representative examples but fewer than in other methods Neural networks can model static (FF) and dynamic (RNN) tasks NN’s are good classifiers BUT –Good representations of data have to be formulated –Training vectors must be statistically representative of the entire input space The use of NN needs a good comprehension of the problem

20. Implementation of Neural Networks Generic architectures (PC’s etc) Specific Neuro-Hardware Dedicated circuits

21. Generic architectures Conventional microprocessors Intel Pentium, Power PC, etc … Advantages –High performances (clock frequency, etc) –Cheap –Software environment available (NN tools, etc) Drawbacks –Too generic, not optimized for very fast neural computations

22. Classification of Hardware NN Hardware –Neurochips Special Purpose General Purpose (Ni1000, L - Neuro) –NeuroComputers Special Purpose (CNAPS, Synapse) General Purpose

23. Specific Neuro-hardware circuits Commercial chips CNAPS, Synapse, etc. Advantages –Closer to the neural applications –High performances in terms of speed Drawbacks –Not optimized to specific applications –Availability –Development tools

24. CNAPS SIMD One instruction sequencing and control unit Processor nodes (PN) Single dimensional array (only right or left nodes)

25. CNAPS 1064

26. CNAPS

27. Dedicated circuits A system where the functionality is buried in the hardware. For specific applications only not changeable Advantages Optimized for a specific application Higher performances than the other systems Drawbacks High development costs in terms of time and money

28. What type of hardware to be used in dedicated circuits ? Custom circuits –ASIC (Application-Specific Integrated Circuit) –Necessity to have good knowledge of the hardware design –Fixed architecture, hardly changeable –Often expensive Programmable logic –Valuable to implement real time systems –Flexibility –Low development costs –Lower performances compared to ASIC (Frequency, etc.)

29. Programmable logic Field Programmable Gate Arrays (FPGAs) –Matrix of logic cells –Programmable interconnection –Additional features (internal memories + embedded resources like multipliers, etc.) –Reconfigurability We can change the configurations as many times as desired

30. Real Time Systems Execution of applications with time constraints. –Hard real-time systems Digital fly-by-wire control system of an aircraft: No lateness is accepted. The lives of people depend on the correct working of the control system of the aircraft. –Soft real-time systems Vending machine: Accept lower performance for lateness, it is not catastrophic when deadlines are not met. It will take longer to handle one client with the vending machine.

31. Real Time Systems ms scale real time system –Connectionist retina for image processing Artificial Retina: combining an image sensor with a parallel architecture µs scale real time system –Level 1 trigger in a HEP experiment

32. Connectionist Retina Integration of a neural network in an artificial retina Screen –Matrix of Active Pixel sensors CAN –8 bits ADC converter 256 levels of grey Processing Architecture –Parallel system where neural networks are implemented Processing Architecture CAN Eye

33. Maharadja Processing Architecture Micro-controller –Generic architecture executing sequential cost with low power consumption Memory –256 Kbytes shared between processor, PE’s, input –Store the network parameters UNE (Unit Neural SIMD –Completely pipelined –16 bit internal data bus) –Processors to compute the neurons outputs –Command bus manages all different operators in UNE Input/Output module –Data acquisition and storage of intermediate results Micro-controller Sequencer Command bus Input/Outputunit Instruction Bus UNE-0UNE-1UNE-2UNE-3 MM M M

34. Level 1 trigger in a HEP experiment High Energy Physics (Particle Physics) Neural networks have provided interesting results as triggers in HEP. –Level 2 : H1 experiment 10 – 20 µs –Level 1 : Dirac experiment 2 µs Particle Recognition High timing constraints (in terms of latency and data throughput)

35. Neural Network architecture …….. 64 128 Execution time : ~500 ns Weights coded in 16 bits States coded in 8 bits with data arriving every BC=25ns 4 Electrons, tau, hadrons, jets

36. Very Fast Architecture TanH PE TanH ACC 256 PE’s Matrix of n*m matrix elements Control unit I/O module TanH are stored in LUTs 1 matrix row computes a neuron The results is back-propagated to calculate the output layer I/O module Control unit

37. PE architecture X AccumulatorMultiplier Weights mem Input data8 16 Addr gen + Data in cmd bus Control Module Data out

38. Neuro-hardware today Generic Real time applications –Microprocessors technology (PCs, computers, i.e. software) is sufficient to implement most of neural applications in real-time (ms or sometimes µs scale) This solution is cheap Very easy to manage Constrained Real time applications –It still remains specific applications where powerful computations are needed e.g. particle physics –It still remains applications where other constraints have to be taken into consideration (Consumption, proximity of sensors, mixed integration, etc.)

39. Clustering Idea : Combine performances of different processors to perform massive parallel computations High speed connection

40. Clustering Advantages –Take advantage of the implicit parallelism of neural networks –Utilization of systems already available (university, Labs, offices, etc.) –High performances : Faster training of a neural net –Very cheap compare to dedicated hardware

41. Clustering Drawbacks –Communications load : Need of very fast links between computers –Software environment for parallel processing –Not possible for embedded applications

42. Hardware Implementations Most real-time applications do not need dedicated hardware implementation –Conventional architectures are generally appropriate –Clustering of generic architectures to combine performances Some specific applications require other solutions –Strong Timing constraints Technology permits to utilize FPGAs –Flexibility –Massive parallelism possible –Other constraints (consumption, etc.) Custom or programmable circuits

43. Questions?