1. Multi-processing and Distributed Systems CS-502 Fall 20071 Multiprocessing and Distributed Systems CS-502 Operating Systems Fall 2007 (Slides include materials from Operating System Concepts, 7 th ed., by Silbershatz, Galvin, & Gagne, Modern Operating Systems, 2 nd ed., by Tanenbaum, and Operating Systems: Internals and Design Principles, 5 th ed., by Stallings)

2. Multi-processing and Distributed Systems CS-502 Fall 20072 Multiprocessing  Distributed Computing (a spectrum) Many independent problems at same time Similar Different One very big problem (or a few) Computations that are physically separated Client-server Inherently dispersed computations Different kinds of computers and operating systems

3. Multi-processing and Distributed Systems CS-502 Fall 20073 Multiprocessing  Distributed Computing (a spectrum) Many independent problems at same time Similar — e.g., banking & credit card; airline reservations Different — e.g., university computer center; your own PC One very big problem (or a few) Computations that are physically separated Client-server Inherently dispersed computations Different kinds of computers and operating systems

4. Multi-processing and Distributed Systems CS-502 Fall 20074 Multiprocessing  Distributed Computing (a spectrum) Many independent problems at same time Similar — e.g., banking & credit card; airline reservations Different — e.g., university computer center; your own PC One very big problem (too big for one computer) Weather modeling, finite element analysis; drug discovery; gene modeling; weapons simulation; etc. Computations that are physically separated Client-server Inherently dispersed computations Different kinds of computers and operating systems

5. Multi-processing and Distributed Systems CS-502 Fall 20075 Multiprocessing  Distributed Computing (a spectrum) Many independent problems at same time Similar — e.g., banking & credit card; airline reservations Different — e.g., university computer center; your own PC One very big problem (too big for one computer) Weather modeling, Finite element analysis; Drug discovery; Gene modeling; Weapons simulation; etc. Computations that are physically separated Client-server Dispersed and/or peer-to-peer –Routing tables for internet –Electric power distribution –International banking Different kinds of computers and operating systems

6. Multi-processing and Distributed Systems CS-502 Fall 20076 Inherently Distributed Computation (example) Internet routing of packets Network layer – send a packet to its IP address Problems No central authority to manage routing in Internet Nodes and links come online or fail rapidly Need to be able to send a packet from any address to any address Solution Distributed routing algorithm

7. Multi-processing and Distributed Systems CS-502 Fall 20077 Distributed routing algorithm (simplified example) Each node “knows” networks other nodes directly connected to it. Each node maintains table of distant networks [network #, 1 st hop, distance] Periodically adjacent nodes exchange tables Update algorithm (for each network in table) If (neighbor’s distance to network + my distance to neighbor < my distance to network), then update my table entry for that network

8. Multi-processing and Distributed Systems CS-502 Fall 20078 Distributed routing algorithm (result) All nodes in Internet have reasonably up-to- date routing tables Rapid responses to changes in network topology, congestion, failures, etc. Very reliable with no central management! Network management software Monitoring health of network (e.g., routing tables) Identifying actual or incipient problems Data and statistics for planning purposes

9. Multi-processing and Distributed Systems CS-502 Fall 20079 Summary Some algorithms are inherently distributed Depend upon computations at physically separate places Result is the cumulative results of individual computations Not much impact on computer or OS architecture

10. Multi-processing and Distributed Systems CS-502 Fall 200710 Taxonomy of Parallel Processing

11. Multi-processing and Distributed Systems CS-502 Fall 200711 Client-Server Computations Very familiar model Parts of computation takes place on User’s PC, credit-card terminal, etc. … and part takes place on central server E.g., updating database, search engine, etc. Central issues Protocols for efficient communication Where to partition the problem

12. Multi-processing and Distributed Systems CS-502 Fall 200712 Very Big Problems Lots and lots of calculations Highly repetitive Easily parallelizable into separate subparts (Usually) dependent upon adjacent subparts Arrays or clusters of computers Independent memories Very fast communication between elements  close coupling

13. Multi-processing and Distributed Systems CS-502 Fall 200713 Closely-coupled Systems Examples –Beowulf clusters – 32-256 PCs –ASCI Red – ~1000-4000 PC-like processors –BlueGene/L – up to 65,536 processing elements

14. Multi-processing and Distributed Systems CS-502 Fall 200714 Taxonomy of Parallel Processing

15. Multi-processing and Distributed Systems CS-502 Fall 200715 IBM BlueGene/L

16. Multi-processing and Distributed Systems CS-502 Fall 200716 Systems for Closely-Coupled Clusters Independent OS for each node E.g., Linux in Beowulf Additional middleware for Distributed process space Synchronization among nodes (Silbershatz, Ch 18) Access to network and peripheral devices Failure management Parallelizing compilers For partitioning a problem into many processes

17. Multi-processing and Distributed Systems CS-502 Fall 200717 Cluster Computer Architecture

18. Multi-processing and Distributed Systems CS-502 Fall 200718 Interconnection Topologies

19. Multi-processing and Distributed Systems CS-502 Fall 200719 Goal Microsecond-level –messages –synchronization and barrier operations among processes

20. Multi-processing and Distributed Systems CS-502 Fall 200720 Questions?

21. Multi-processing and Distributed Systems CS-502 Fall 200721 Multiprocessing  Distributed Computing (a spectrum) Many independent problems at same time Similar — e.g., banking & credit card; airline reservations Different — e.g., university computer center; your own PC One very big problem (or a few) Problems that physically separated by their very nature Different kinds of computers and operating systems

22. Multi-processing and Distributed Systems CS-502 Fall 200722 Many Separate Computations Similar – banking & credit card; airline reservation; … Transaction processing Thousands of small transactions per second Few (if any) communications among transactions –Exception: locking, mutual exclusion, serialization Common database

23. Multi-processing and Distributed Systems CS-502 Fall 200723 Requirements Lots of disks, enough CPU power Accessible via multiple paths High availability Fault-tolerant components & OS support Processor independence Any transaction on any processor Global file system Load-balancing Scalable

24. Multi-processing and Distributed Systems CS-502 Fall 200724 Options Cluster (of a somewhat different sort) Shared-memory multi-processor

25. Multi-processing and Distributed Systems CS-502 Fall 200725 Taxonomy of Parallel Processing

26. Multi-processing and Distributed Systems CS-502 Fall 200726 Cluster Computing Scalable to large number of processors Common I/O space –Multiple channels for disk access –Hyper-channel, Fiber-channel, etc. Multi-headed disks –At least two paths from any processor to any disk Remote Procedure Call for communication among processors –DCOM, CORBA, Java RMI, etc.

27. Multi-processing and Distributed Systems CS-502 Fall 200727 Shared-Memory Multiprocessor All processors can access all memory Some memory may be faster than other memory Any process can run on any processor Multiple processors executing in same address space concurrently Multiple paths to disks (as with cluster)

28. Multi-processing and Distributed Systems CS-502 Fall 200728 Shared Memory Multiprocessors Bus-based Bus contention limits the number of CPUs Lower bus contention Caches need to be synced (big deal) Compiler places data and text in private or shared memory

29. Multi-processing and Distributed Systems CS-502 Fall 200729 Multiprocessors (2) - Crossbar Can support a large number of CPUs - Non-blocking network Cost/performance effective up to about 100 CPU – growing as n 2

30. Multi-processing and Distributed Systems CS-502 Fall 200730 Multiprocessors(3) – Multistage Switching Networks Omega Network – blocking –Lower cost, longer latency –For N CPUs and N memories – log 2 n stages of n/2 switches

31. Multi-processing and Distributed Systems CS-502 Fall 200731 Type of Multiprocessors – UMA vs. NUMA UMA (Uniform Memory Access) –All memory is equal –Familiar programming model –Number of processors is limited 8-32 –Symmetrical No processor is different from others NUMA (Non-Uniform Memory Access) –Single address space visible to all CPUs –Access to remote memory via commands LOAD & STORE remote memory access slower than to local

32. Multi-processing and Distributed Systems CS-502 Fall 200732 Common SMP Implementations Multiple processors on same board or bus Dual core and multi-core processors I.e., two or more processors on same chip Share same L2 cache Hyperthreading One processor shares its circuitry among two threads or processes Separate registers, PSW, etc. Combinations of above

33. Multi-processing and Distributed Systems CS-502 Fall 200733 Hardware issue – Cache Synchronization Each processor cache monitors memory accesses on bus (i.e., “snoops”) –Memory updates are propagated to other caches –Complex cache management hardware –Order of operations may be ambiguous Some data must be marked as not cacheable –Visible to programming model

34. Multi-processing and Distributed Systems CS-502 Fall 200734 Operating System Considerations Master-slave vs. full symmetry Explicit cache flushing When page is replaced or invalidated Processor affinity of processes Context switches across processors can be expensive Synchronization across processors Interrupt handling

35. Multi-processing and Distributed Systems CS-502 Fall 200735 Multiprocessor OS – Master-Slave One CPU (master) runs the OS and applies most policies Other CPUs –run applications –Minimal OS to acquire and terminate processes Relatively simple OS Master processor can become a bottleneck for a large number of slave processors

36. Multi-processing and Distributed Systems CS-502 Fall 200736 Multiprocessor OS – Symmetric Multi-Processor (SMP) Any processor can execute the OS and any applications Synchronization within the OS is the issue 1.Lock the whole OS – poor utilization – long queues waiting to use OS 2.OS critical regions – much preferred –Identify independent OS critical regions that be executed independently – protect with mutex –Identify independent critical OS tables – protect access with MUTEX –Design OS code to avoid deadlocks –The art of the OS designer –Maintenance requires great care

37. Multi-processing and Distributed Systems CS-502 Fall 200737 Multiprocessor OS – SMP (continued) Multiprocessor Synchronization –Need special instructions – test-and-set –Spinlocks are common Can context switch if time in critical region is greater than context switch time OS designer must understand the performance of OS critical regions Context switch time could be onerous –Data cached on one processor needs to be re- cached on another

38. Multi-processing and Distributed Systems CS-502 Fall 200738 Multiprocessor Scheduling When processes are independent (e.g., timesharing) –Allocate CPU to highest priority process –Tweaks For a process with a spinlock, let it run until it releases the lock To reduce TLB and memory cache flushes, try to run a process on the same CPU each time it runs For groups of related processes –Attempt to simultaneously allocate CPUs to all related processes (space sharing) –Run all threads to termination or block –Gang schedule – apply a scheduling policy to related processes together

39. Multi-processing and Distributed Systems CS-502 Fall 200739 Linux SMP Support Multi-threaded kernel Any processor can handle any interrupt Interrupts rarely disabled, and only for shortest time possible Multiple processors can be active in system calls concurrently Process vs. Interrupt context Process context:– when processor is acting on behalf of a specific process Interrupt context:– when processor is acting on behalf of no particular process Extensive use of spinlocks Processor affinity properties in task_struct

40. Multi-processing and Distributed Systems CS-502 Fall 200740 WPI Computing & Communications Center CCC cluster – 11 dual 2.4 GHz Xeon processors toth – 16-node 1.5 GHz Itanium2 big16 – 8 dual-core 2.4 GHz Opterons toaster – NetApp 5.5 TByte Fiber Channel RAID NFS breadbox – NetApp 12 TByte RAID NFS Many others All connected together by very fast networks

41. Multi-processing and Distributed Systems CS-502 Fall 200741 Questions?