1. Neural Network Implementations on Parallel Architectures

2. Index Neural Networks Learning in NN Parallelism Characteristics
Mapping Schemes & Architectures

3. Artificial Neural Networks
inspired from human brain
parallel,distributed computing model
consists of a large number of simple,neuron-like processing elements called units
weighted,directed connections between pairs of units.

4. Artificial Neural Networks-2
weights may be positive or negative
each unit computes a simple function of its inputs,which are weighted outputs from other units

5. Artificial Neural Networks-3
a threshold value is used in each neuron to determine the activation of the output
learning in NN:
finding the weights and threshold values
training set
multi-layer,feedforward networks: input layer,hidden layer,output layer

6. Learning in ANN initialize all weights apply input vector to network
propagate vector forward and obtain unit outputs
compare output layer response with desired outputs
compute and propagate error measure backward, correcting
weights layer by layer
iIterate until ”good” mapping is achieved

7. Learning in ANN-2

8. Parallelism further speed-up of training
neural networks exhibit high degree of parallelism(distributed set of units operating simultaneously)
process of parallelism:
what type of machine?
how to parallelize?

9. Parallelism-2 different neural network models
highly dependend on the model used
SIMD(small computation & a lot of data exchange)
one neuron for one processor
MIMD (distributed memory & message passing)
bad performance in frequent communication

10. Characteristics theoretical analysis of the inherent algorithm
portability
ease of use
access to ANN model description

11. Historical Data Integration
prediction of the sensor output
two parallelism methods:
parallel calculation of weighted sum
time increases with the number of processors
parallel training of each seperate NN
time decreases with the number of processors
8 RISC processors
4 MB cache memory
512 RAM

12. Method-1

13. Method-2

14. Distributed Training Data

15. A library on MIMD machines
distributed shared memory

16. A library on MIMD machines-2
several communication and syncronization schemes
message passing or shared memory
thread programming with shared memory has the best performance
every data is shared but handled only by one
processor
training of a Kohonen map of 100*100 neurons with iterations with 8 processors are 7 times faster than the sequential execution.

17. A library on MIMD machines-3

18. AP1000 Architecture

19. AP1000 Architecture-2 MIMD computer with distributed memory
vertical slicing of the network
3 methods for communication
One to one communication
Rotated messages in horizontal and vertical rings
Parallel routed messages
different neural network implementations

20. AP1000 Architecture-3
different mappings according to the network and the training data
heuristic on training time
combine multiple degrees of parallelism
training set parallelism
node parallelism
pipelining parallelism

21. References
“APPROACH TO PARALLEL TRAINING OF INTEGRATION HISTORICAL DATA NEURAL NETWORKS”, V. TURCHENKO1, C. TRIKI2, A. SACHENKO1
“A LIBRARY TO IMPLEMENT NEURAL NETWORKS ON MIMD MACHINES”,Y.BONIFACE, F.ALEXANDRE,S. VIALLE
“A BRIDGE BETWEEN TWO PARADIGMS FOR PARALLELISM: NEURAL NETWORKS AND GENERAL PURPOSE MIMD COMPUTERS”, Y.BONIFACE, F.ALEXANDRE,S. VIALLE
“PARALLEL ENVIRONMENTS FOR IMPLEMENTING NEURAL NETWORKS”, M. MISRA