1. MIMD Distributed Memory Architectures message-passing multicomputers

2. MIMD-DM organization Each node includes –full processor (control and ALU) –memory –connection to interconnect network Typically commodity processors, memory Value in interconnect –high speed, high bandwidth

3. MIMD-DM CPUM e m Comm Node CPUM e m Comm Node CPUM e m Comm Node Network

4. MIMD-DM Issues Connection Network –fast –high bandwidth –scalable Communications –explicit message passing –parallel languages Occam 2, variations of C, Pascal –libraries for sequential languages PVM, MPI, Java with CSP

5. Message Passing Point-to-Point Requires explicit commands in program –Send, Receive Must be synchronized among different processors –Sends and Receives must match –Avoid Deadlock -- all processors waiting, none able to communicate Multi-processor communications –e.g. broadcast, reduce

6. Deadlock Send

7. Message Passing Systems PVM Parallel Virtual Machine –developed at national lab –intended for use with local area networks –adapted for most MIMD parallel computers IBM SP2, Cray T3E, SGI Origin –Provides library of function calls for C or FORTRAN Send, Receive, broadcast, reduce message packing/unpacking synchronization

8. Message Passing Systems MPI Message Passing Interface –developed by consortium of vendors, users, labs –intended to replace proprietary systems, PVM thus providing portability –takes best ideas from several systems –adapted for most MIMD parallel computers IBM SP2, Cray T3E, SGI Origin –Provides library of function calls for C or FORTRAN Send, Receive, broadcast, reduce message packing/unpacking synchronization

9. Message Passing Systems Occam 2 –full parallel language –co-developed with processor, Inmos Transputer –provides parallelism within and among processors –uses CSP model Communicating Sequential Processes Developed by Anthony Hoare explicit point-to-point channels for communications –no longer imp[ortant transputer fell behind in development race

10. Message Passing Systems Java with CSP –intended for concurrent and parallel computing in Java –Based on CSP / Occam 2 model –Provides processes, channels, in Java for a single processor between processors –Processor-to-processor channels developed at Colgate

11. Interconnection Network Speed and Bandwidth are critical Low cost networks –local area network (ethernet, token ring) –can be set up with packages PVM MPI High Speed Networks –The heart of a MIMD-DM Parallel Machine