1. Convergence of Parallel Architectures

2. History
Historically, parallel architectures tied to programming models
Divergent architectures, with no predictable pattern of growth.
Convergence of Parallel Architectures
Application Software
System
Software
Systolic
Arrays
SIMD
Architecture
Tied = zviazaný
Uncertainity = neistota
Message Passing
Dataflow
Shared Memory
Uncertainty of direction paralyzed parallel software development!

3. Today
Extension of “computer architecture” to support communication and cooperation
OLD: Instruction Set Architecture
NEW: Communication Architecture
Defines
Critical abstractions, boundaries, and primitives (interfaces)
Organizational structures that implement interfaces (hw or sw)
Compilers, libraries and OS are important bridges today
Convergence of Parallel Architectures

4. Modern Layered Framework
Convergence of Parallel Architectures
CAD
Multipr
ogramming
Shar ed
addr
ess
Message
passing
Data
parallel
Database
Scientific modeling
Parallel applications Pr
ogramming models
Communication abstraction
User/system boundary
Compilation
or library
Operating systems support
Communication har
dwar e
Physical communication medium
Har
e/softwar
e boundary

5. Programming Model What programmer uses in coding applications
Specifies communication and synchronization
Examples:
Multiprogramming: no communication or synch. at program level
Shared address space: like bulletin board
Message passing: like letters or phone calls, explicit point to point
Data parallel: more regimented, global actions on data
Implemented with shared address space or message passing
Convergence of Parallel Architectures

6. Communication Abstraction
User level communication primitives provided
Realizes the programming model
Mapping exists between language primitives of programming model and these primitives
Supported directly by hw, or via OS, or via user sw
Lot of debate about what to support in sw and gap between layers
Today:
Hw/sw interface tends to be flat, i.e. complexity roughly uniform
Compilers and software play important roles as bridges today
Technology trends exert strong influence
Result is convergence in organizational structure
Relatively simple, general purpose communication primitives
Convergence of Parallel Architectures
Roughly = približne
Exert = vyvíjať

7. Communication Architecture
= User/System Interface + Implementation
User/System Interface:
Comm. primitives exposed to user-level by hw and system-level sw
Implementation:
Organizational structures that implement the primitives: hw or OS
How optimized are they? How integrated into processing node?
Structure of network
Goals:
Performance
Broad applicability
Programmability
Scalability
Low Cost
Convergence of Parallel Architectures

8. Evolution of Architectural Models
Historically machines tailored to programming models
Prog. model, comm. abstraction, and machine organization lumped together as the “architecture”
Evolution helps understand convergence
Identify core concepts
Shared Address Space
Message Passing
Data Parallel
Others:
Dataflow
Systolic Arrays
Examine programming model, motivation, intended applications, and contributions to convergence
Convergence of Parallel Architectures

9. Shared Address Space (SAS) Architectures
Any processor can directly reference any memory location
Communication occurs implicitly as result of loads and stores
Convenient:
Location transparency
Similar programming model to time-sharing on uniprocessors
Except processes run on different processors
Good throughput on multiprogrammed workloads
Naturally provided on wide range of platforms
History dates at least to precursors of mainframes in early 60s
Wide range of scale: few to hundreds of processors
Popularly known as shared memory machines/processors or model
Ambiguous: memory may be physically distributed among processors
Convergence of Parallel Architectures

10. SAS Model
Process: virtual address space plus one or more threads of control
Portions of address spaces of processes are shared S t o r e P 1 2 n L a d p i v
Virtual address spaces for a
collection of processes communicating
via shared addresses
Machine physical address space
Shared portion
of address space
Private portion
Common physical
addresses
Convergence of Parallel Architectures
Writes to shared address visible to other threads (in other processes too)
Natural extension of uniprocessors model: conventional memory operations for comm.; special atomic operations for synchronization
OS uses shared memory to coordinate processes

11. Communication Hardware
Also natural extension of uniprocessor
Already have processor, one or more memory modules and I/O controllers connected by hardware interconnect of some sort
Convergence of Parallel Architectures
Memory capacity increased by adding modules, I/O by controllers
Add processors for processing!
For higher-throughput multiprogramming, or parallel programs

12. History Motivated by multiprogramming
“Mainframe” approach
Motivated by multiprogramming
Extends crossbar used for mem bw and I/O
Originally processor cost limited to small
later, cost of crossbar
Bandwidth scales with p
High incremental cost; use multistage instead
“Minicomputer” approach
Almost all microprocessor systems have bus
Motivated by multiprogramming, TP
Used heavily for parallel computing
Called symmetric multiprocessor (SMP)
Latency larger than for uniprocessor
Bus is bandwidth bottleneck
caching is key: coherence problem
Low incremental cost
Convergence of Parallel Architectures

13. Example: Intel Pentium Pro Quad
Convergence of Parallel Architectures
All coherence and multiprocessing glue in processor module
Highly integrated, targeted at high volume
Low latency and bandwidth

14. Example: SUN Enterprise
System Bus Architecture — Jupiter Interconnect
SPARC Enterprise M3000, M4000, M5000, M8000, and M9000 servers utilize a system interconnect designed to deliver massive bandwidth and consistent low latency between components.
The SPARC Enterprise M5000 server is implemented within a single motherboard but features two logical system boards.
Similar to the SPARC Enterprise M4000 server design, each logical system board contains two system controllers that connect to each other, as well as CPU modules, memory access controllers, and an IOU. In addition, each system controller connects to a corresponding system controller on the other logical system board
Convergence of Parallel Architectures

15. SPARC Enterprise M5000 server interconnect diagram
Convergence of Parallel Architectures *

16. Charakteristics of SMP Machines
All processors see everything
Memory, I/O, interrupts, etc.
There is only one kernel
The scheduler determines which applications are assigned to which processor
Application can migrate between processors
They do not typically share caches
Although hybrid cache architectures are on the way
Convergence of Parallel Architectures

17. Example SMP Motherboard
Quad-CPU AMD Opteron support
Convergence of Parallel Architectures

18. Problems with SMP
SMP system can be complex to set up and maintain because of duplicated HW such as processor fans, etc.
They tend to be noisy as well due to fans
SMP does not scale perfectly
Because the memory is shared can develop “hot spots” where multiple applications must serialize on a single piece of data
Process migration can lead to poor cache utilization
We need to flush the cache if a process migrates
Multiple processors can lead to race conditions
We need to provide for multi-processor synchronization
Convergence of Parallel Architectures

19. Scaling Up Problem is interconnect: cost (crossbar) or bandwidth (bus)
“Dance hall”
Distributed memory
Convergence of Parallel Architectures
Problem is interconnect: cost (crossbar) or bandwidth (bus)
Dance-hall: bandwidth still scalable, but lower cost than crossbar
latencies to memory uniform, but uniformly large
Distributed memory or non-uniform memory access (NUMA)
Construct shared address space out of simple message transactions across a general-purpose network (e.g. read-request, read-response)
Caching shared (particularly nonlocal) data?

20. Example: Cray T3E Scale up to 1024 processors, 480MB/s links
Convergence of Parallel Architectures
Scale up to 1024 processors, 480MB/s links
Memory controller generates comm. request for nonlocal references
No hardware mechanism for coherence (SGI Origin etc. provide this)

21. SMP vs NUMA
Convergence of Parallel Architectures

22. NUMA Opteron’s Hammer architecture
Because each chip has its own memory resources and does not have to fight for a single unified pool of memory, Opteron has what is known as a "NUMA" (Non-Unified Memory Architecture) scheme.
The benefits of Opteron’s NUMA design will be most clearly seen in n-way systems.
Convergence of Parallel Architectures

23. Multicore Is Bad News For Supercomputers
With no other way to improve the performance of processors further, chip makers have staked their future on putting more and more processor cores on the same chip.
Sandia National Laboratories, in New Mexico,
Simulated future high-performance computers containing the 8-core, 16''core, and 32-core microprocessors that chip makers say are the future of the industry.
The results are distressing. Because of limited memory bandwidth and memory-management schemes that are poorly suited to supercomputers, the performance of these machines would level off or even decline with more cores. The performance is especially bad for informatics applications— data-intensive programs that are increasingly crucial to the labs’ national security function.
Convergence of Parallel Architectures

24. Adding cores slows data-intensive applications
High-performance computing has historically focused on solving differential equations describing physical systems, such as Earth’s atmosphere or a hydrogen bomb’s fission trigger. These systems lend themselves to being divided up into grids, so the physical system can, to a degree, be mapped to the physical location of processors or processor cores, thus minimizing delays in moving data. For informatics, more cores doesn’t mean better performance [see red line in ”Trouble Ahead”], according to Sandia’s simulation. ”After about 8 cores, there’s no improvement,” says James Peery, director of computation, computers, information, and mathematics at Sandia. ”At 16 cores, it looks like 2.”
Convergence of Parallel Architectures

25. Multicore Is Bad News For Supercomputers
Trouble ahead
Convergence of Parallel Architectures

26. Heart of the trouble
At the heart of the trouble is the so-called memory wall—the growing disparity between how fast a CPU can operate on data and how fast it can get the data it needs.
Although the number of cores per processor is increasing, the number of connections from the chip to the rest of the computer is not.
So keeping all the cores fed with data is a problem.
In informatics applications, the problem is worse because there is no physical relationship between what a processor may be working on and where the next set of data it needs may reside.
Instead of being in the cache of the core next door, the data may be on a DRAM chip in a rack 20 meters away and need to leave the chip, pass through one or more routers and optical fibers, and find its way onto the processor.
Convergence of Parallel Architectures

27. Solution (?)
“The key to solving this bottleneck is tighter, and maybe smarter, integration of memory and processors,”
says Peery (James Peery, director of computation, computers, information, and mathematics at Sandia).
For its part, Sandia is exploring the impact of stacking memory chips atop processors to improve memory bandwidth.
Convergence of Parallel Architectures

28. The rise (and limit) of Many-Core
Convergence of Parallel Architectures

29. Message Passing Architectures
Complete computer as building block, including I/O
Communication via explicit I/O operations
Programming model: directly access only private address space (local memory), comm. via explicit messages (send/receive)
High-level block diagram similar to NUMA shred memory approach
But comm. integrated at IO level, needn’t be into memory system
Like networks of workstations (clusters), but tighter integration
Programming model more removed from basic hardware operations
Library or OS intervention
Convergence of Parallel Architectures

30. Message-Passing AbstractionPr
ocess P Q
Addr
ess Y X
Send
X, Q, t
Receive , t
Match
Local pr
addr
ess space
Convergence of Parallel Architectures
Send specifies buffer to be transmitted and receiving process
Recv specifies sending process and application storage to receive into
Memory to memory copy, but need to name processes
Optional tag on send and matching rule on receive
User process names local data and entities in process/tag space too
In simplest form, the send/recv match achieves pairwise synch event
Other variants too
Many overheads: copying, buffer management, protection

31. Evolution of Message-Passing Machines
Early machines: FIFO on each link
Hw close to prog. Model; synchronous ops
Replaced by DMA, enabling non-blocking ops
Buffered by system at destination until recv
Diminishing role of topology
Store&forward routing: topology important
Introduction of pipelined routing made it less so
Cost is in node-network interface
Simplifies programming
Convergence of Parallel Architectures

32. Example: IBM SP-2 Made out of essentially complete RS6000 workstations
Convergence of Parallel Architectures
Made out of essentially complete RS6000 workstations
Network interface integrated in I/O bus (bw limited by I/O bus)

33. Example Intel Paragon
Convergence of Parallel Architectures

34. IBM’s Hydro Cluster
IBM's new Power 575 supercomputer uses a new system of chip-level water-cooling to keep its processors chilled. Nicknamed "Hydro Cluster", the machine actually uses 448 of the new 5GHz POWER6 processors.
Convergence of Parallel Architectures

35. Toward Architectural Convergence
Evolution and role of software have blurred boundary
Traditional MP operations (send/recv) are supported on most shared memory machines via buffers
Send involves writing data into the buffer
Receive involves reading the data from shared storage
Flags or locks are used to control access to the buffer and indicate events such as message arrival
On a MP machine, a user process may construct a global address space
Access to such a global address can be performed in SW through an explicit message transaction.
Most MP libraries allow a process to accept a message for any process, so each process can serve data request from the others.
Convergence of Parallel Architectures

36. Toward Architectural Convergence
Evolution and role of software have blurred boundary
On a MP machine, a user process may construct a global address space
A logical read is realized by sending a request to the process containing the object and receiving a response.
The actual message transaction may be hidden from the user; it may be carried out by compiler-generated code for access to a shared variable.
A shared virtual address space can be established on a message- passing machine at the page level.
A collection of processes has a region of shared address but, for each process, only the pages that are local to it are accessible.
Upon access to a missing (i.e. remote) page, a page fault occurs and the OS engages the remote node in a message transaction to transfer the page and map it into the user address sapce
Convergence of Parallel Architectures

37. Toward Architectural Convergence
Evolution and role of software have blurred boundary
Send/recv supported on shared memory machines via buffers
Can construct global address space on MP using hashing
Page-based (or finer-grained) shared virtual memory
Hardware organization converging too
Tighter network interface (NI) integration even for MP (low- latency, high-bandwidth)
Even clusters of workstations/SMPs are parallel systems
Emergence of fast system area networks (SAN)
Programming models distinct, but organizations converging
Nodes connected by general network and communication assists
Implementations also converging, at least in high-end machines
Convergence of Parallel Architectures

38. Data Parallel Systems Programming model Architectural model
Operations performed in parallel on each element of data structure
Logically single thread of control, performs sequential or parallel steps
Conceptually, a processor associated with each data element
Architectural model
Array of many simple, cheap processors with little memory each
Processors don’t sequence through instructions
Attached to a control processor that issues instructions
Specialized and general communication, cheap global synchronization
Original motivations
Matches simple differential equation solvers
Centralize high cost of instruction fetch/sequencing
Convergence of Parallel Architectures

39. Evolution and Convergence
Rigid control structure (SIMD in Flynn taxonomy)
SISD = uniprocessor, MIMD = multiprocessor
Popular when cost savings of centralized sequencer high
60s when CPU was a cabinet
Replaced by vectors in mid-70s
More flexible w.r.t. memory layout and easier to manage
Revived in mid-80s when 32-bit datapath slices just fit on chip
No longer true with modern microprocessors
Other reasons for demise
Simple, regular applications have good locality, can do well anyway
Loss of applicability due to hardwiring data parallelism
MIMD machines as effective for data parallelism and more general
Prog. model converges with SPMD (single program multiple data)
Contributes need for fast global synchronization
Structured global address space, implemented with either SAS or MP
Convergence of Parallel Architectures

40. Dataflow Architectures
Represent computation as a graph of essential dependences
Logical processor at each node, activated by availability of operands
Message (tokens) carrying tag of next instruction sent to next processor
Tag compared with others in matching store; match fires execution
Convergence of Parallel Architectures

41. Evolution and Convergence
Key characteristics
Ability to name operations, synchronization, dynamic scheduling
Problems
Operations have locality across them, useful to group together
Handling complex data structures like arrays
Complexity of matching store and memory units
Expose too much parallelism (?)
Converged to use conventional processors and memory
Support for large, dynamic set of threads to map to processors
Typically shared address space as well
But separation of progr. model from hardware (like data-parallel)
Lasting contributions:
Integration of communication with thread (handler) generation
Tightly integrated communication and fine-grained synchronization
Remained useful concept for software (compilers etc.)
Convergence of Parallel Architectures

42. Systolic Architectures
Replace single processor with array of regular processing elements
Orchestrate data flow for high throughput with less memory access
Convergence of Parallel Architectures
Different from pipelining
Nonlinear array structure, multidirection data flow, each PE may have (small) local instruction and data memory
Different from SIMD: each PE may do something different
Initial motivation: VLSI enables inexpensive special-purpose chips
Represent algorithms directly by chips connected in regular pattern

43. Systolic Arrays (contd.)
Example: Systolic array for 1-D convolution y ( i
) = w 1 x
) + 2
+ 1) + 3
+ 2) + 4
+ 3) 8 7 6 5 in
out
out =
in + =
Convergence of Parallel Architectures
Practical realizations (e.g. iWARP) use quite general processors
Enable variety of algorithms on same hardware
But dedicated interconnect channels
Data transfer directly from register to register across channel
Specialized, and same problems as SIMD
General purpose systems work well for same algorithms (locality etc.)

44. Convergence: Generic Parallel Architecture
A generic modern multiprocessor
Convergence of Parallel Architectures
Node: processor(s), memory system, plus communication assist
Network interface and communication controller
Scalable network
Convergence allows lots of innovation, now within framework
Integration of assist with node, what operations, how efficiently...

45. Fundamental Design Issues

46. Understanding Parallel Architecture
Traditional taxonomies – SIMD/MIMD (multiple general-purpose processors)
Programming models not enough, nor hardware structures
We cannot focus entirely on programming models since in many cases widely differing machine organizations support a common programming model.
We cannot just look at HW structures either, since common elements are employed in many different ways.
Instead, we ought to focus our attention on the architectural distinctions that make a difference to the software that is to run on the machine.
Fundamental Design Issues

47. Understanding Parallel Architecture
Programming models not enough, nor hardware structures
In particular, we need to highlight those aspects that influence how a compiler should generate code from a high-level parallel language, how a library writer would code a well optimized library, or how an application would be written in a low-level parallel language.
We can then approach the design problem as one that is constrained from above by how programs use the machine and from below by what the basic technology can provide.
Fundamentally, we must understand the operations that are provided at the user-level communication abstraction, how various programming models are mapped to these primitives, and how these primitives are mapped to the actual hardware.
Fundamental Design Issues

48. Understanding Parallel Architecture
The communication abstraction forms the key interface between the programming model and the system implementation.
Viewed from the SW side, it must have a precise, well-defined meaning so that the same program will run correctly on many implementations.
The operations provided at this layer must be simple, composable entities with clear costs, so that the software can be optimized for performance.
Viewed from the hardware side, it also must have a well defined meaning so that the machine designer can determine where performance optimizations can be performed without violating the software assumptions.
Fundamental Design Issues

49. Understanding Parallel Architecture
The communication abstraction forms the key interface between the programming model and the system implementation.
While the abstraction needs to be precise, the machine designer would like it not to be overly specific, so it does not prohibit useful techniques for performance enhancement or frustrate efforts to exploit properties of newer technologies
The communication abstraction is, in effect, a contract between the HW and the SW allowing each the flexibility to improve what it does, while working correctly together.
To understand the “terms” of this contract, we need to look more carefully at the basic requirements of a programming model.
Fundamental Design Issues

50. Understanding Parallel Architecture
A parallel program consists of one or more threads of control operating on data.
A parallel programming model specifies
what data can be named by the threads,
what operations can be performed on the named data,
and what ordering exists among these operations.
Fundamental Design Issues

51. Fundamental Design Issues
At any layer, interface (contract) aspect and performance aspects
Naming: How are logically shared data and/or processes referenced?
Operations: What operations are provided on these data
Ordering: How are accesses to data ordered and coordinated?
Replication: How are data replicated to reduce communication?
Communication Cost: Latency, bandwidth, overhead, occupancy
Understand at programming model first, since that sets requirements
Other issues
• Node Granularity: How to split between processors and memory?
• ...

52. Understanding Parallel Architecture
To make these issues concrete, consider the programming model for a uniprocessor.
A thread can name the locations in its virtual address space and can name machine registers.
In some systems the address space is broken up into distinct code, stack, and heap segments, while in others it is flat.
Similarly, different programming languages provide access to the address space in different ways;
for example, some allow pointers and dynamic storage allocation, while others do not.
Regardless of these variations, the instruction set provides the operations that can be performed on the named locations.
Fundamental Design Issues

53. Understanding Parallel Architecture
To make these issues concrete, consider the programming model for a uniprocessor.
For example, in RISC machines the thread can load data from or store data to memory, but perform arithmetic and comparisons only on data in registers.
Older instruction sets support arithmetic on either.
Compilers typically mask these differences at the hardware/ software boundary, so the user’s programing model is one of performing operations on variables which hold data.
The hardware translates each virtual address to a physical address on every operation.
The ordering among memory operations is sequential program order.
Fundamental Design Issues

54. Sequential program order
The programmer’s view is that variables are read and modified in the top-to-bottom, left-to-right order specified in the program.
More precisely, the value returned by a read to an address is the last value written to the address in the sequential execution order of the program.
This ordering assumption is essential to the logic of the program.
However, the reads and writes may not actually be performed in program order, because the compiler performs optimizations when translating the program to the instruction set and the hardware performs optimizations when executing the instructions.
Both make sure the program cannot tell that the order has been changed.
Fundamental Design Issues

55. Sequential program order
The compiler and hardware preserve the dependence order.
If a variable is written and then read later in the program order, they make sure that the later operation uses the proper value.
Collections of reads with no intervening writes may be completely reordered
Writes to different addresses can be reordered as long as dependences from intervening reads are preserved.
This reordering occurs at the compilation level, for example, when the compiler allocates variables to registers, manipulates expressions to improve pipelining, or transforms loops to reduce overhead and improve the data access pattern.
It occurs at the machine level when instruction execution is pipelined, multiple instructions are issued per cycle, or when write buffers are used to hide memory latency.
Fundamental Design Issues

56. Sequential Programming Model
Fundamental Design Issues
Contract
Naming: Can name any variable in virtual address space
Hardware (and perhaps compilers) does translation to physical addresses
Operations: Loads and Stores
Ordering: Sequential program order
Performance
Rely on dependences on single location (mostly): dependence order
Compilers and hardware violate other orders without getting caught
Compiler: reordering and register allocation
Hardware: out of order, pipeline bypassing, write buffers
Transparent replication in caches

57. SAS Programming Model
Naming: Any process can name any variable in shared space
Operations: loads and stores, plus those needed for ordering
Simplest Ordering Model:
Within a process/thread: sequential program order
Across threads: some interleaving (as in time-sharing)
Additional orders through synchronization
Again, compilers/hardware can violate orders without getting caught
Different, more subtle ordering models also possible (discussed later)
Fundamental Design Issues
SAS = shared address space

58. Synchronization Mutual exclusion (locks) Event synchronization
Ensure certain operations on certain data can be performed by only one process at a time
Room that only one person can enter at a time
No ordering guarantees
Event synchronization
Ordering of events to preserve dependences
e.g. producer —> consumer of data
3 main types:
point-to-point
global
group
Fundamental Design Issues

59. Message Passing Programming Model
Naming: Processes can name private data directly.
No shared address space
Operations: Explicit communication through send and receive
Send transfers data from private address space to another process
Receive copies data from process to private address space
Must be able to name processes
Ordering:
Program order within a process
Send and receive can provide pt to pt synch between processes
Mutual exclusion inherent
Can construct global address space:
Process number + address within process address space
But no direct operations on these names
Fundamental Design Issues

60. Naming and Operations
Fundamental Design Issues
Naming and operations in programming model can be directly supported by lower levels, or translated by compiler, libraries or OS
Example: Shared virtual address space in programming model
Hardware interface supports shared physical address space
Direct support by hardware through v-to-p mappings, no software layers
Hardware supports independent physical address spaces
Can provide SAS through OS, so in system/user interface
virtual-to-physical mappings only for data that are local
remote data accesses incur page faults; brought in via page fault handlers
same programming model, different hardware requirements and cost model
Or through compilers or runtime, so above sys/user interface
shared objects, instrumentation of shared accesses, compiler support

61. Naming and Operations (contd)
Example: Implementing Message Passing
Direct support at hardware interface
But match and buffering benefit from more flexibility
Support at sys/user interface or above in software (almost always)
Hardware interface provides basic data transport (well suited)
Send/receive built in sw for flexibility (protection, buffering)
Choices at user/system interface:
OS each time: expensive
OS sets up once/infrequently, then little sw involvement each time
Or lower interfaces provide SAS, and send/receive built on top with buffers and loads/stores
Need to examine the issues and tradeoffs at every layer
Frequencies and types of operations, costs
Fundamental Design Issues

62. Ordering
Message passing: no assumptions on orders across processes except those imposed by send/receive pairs
SAS: How processes see the order of other processes’ references defines semantics of SAS
Ordering very important and subtle
Uniprocessors play tricks with orders to gain parallelism or locality
These are more important in multiprocessors
Need to understand which old tricks are valid, and learn new ones
How programs behave, what they rely on, and hardware implications
Fundamental Design Issues

63. Replication Very important for reducing data transfer/communication
Again, depends on naming model
Uniprocessor: caches do it automatically
Reduce communication with memory
Message Passing naming model at an interface
A receive replicates, giving a new name; subsequently use new name
Replication is explicit in software above that interface
SAS naming model at an interface
A load brings in data transparently, so can replicate transparently
Hardware caches do this, e.g. in shared physical address space
OS can do it at page level in shared virtual address space, or objects
No explicit renaming, many copies for same name: coherence problem
in uniprocessors, “coherence” of copies is natural in memory hierarchy
Fundamental Design Issues

64. Communication Performance
Performance characteristics determine usage of operations at a layer
Programmer, compilers etc make choices based on this
Fundamentally, three characteristics:
Latency: time taken for an operation
Bandwidth: rate of performing operations
Cost: impact on execution time of program
If processor does one thing at a time: bandwidth µ 1/latency
But actually more complex in modern systems
Characteristics apply to overall operations, as well as individual components of a system, however small
We’ll focus on communication or data transfer across nodes
Fundamental Design Issues

65. Simple Example Component performs an operation in 100ns
Simple bandwidth: 10 Mops
Internally pipeline depth 10 => bandwidth 100 Mops
Rate determined by slowest stage of pipeline, not overall latency
Delivered bandwidth on application depends on initiation frequency
Suppose application performs 100 M operations. What is cost?
op count * op latency gives 10 sec (upper bound)
op count / peak op rate gives 1 sec (lower bound)
assumes full overlap of latency with useful work, so just issue cost
if application can do 50 ns of useful work before depending on result of op, cost to application is the other 50ns of latency
Fundamental Design Issues

66. Linear Model of Data Transfer Latency
Transfer time (n) = T0 + n/B
useful for message passing, memory access, vector ops etc
As n increases, bandwidth approaches asymptotic rate B
How quickly it approaches depends on T0
Size needed for half bandwidth (half-power point):
n1/2 = T0 / B
But linear model not enough
When can next transfer be initiated? Can cost be overlapped?
Need to know how transfer is performed
Fundamental Design Issues

67. Communication Cost Model
Comm Time per message= Overhead + Assist Occupancy + Network Delay + Size/Bandwidth + Contention
= ov + oc + l + n/B + Tc
Overhead and assist occupancy may be f(n) or not
Each component along the way has occupancy and delay
Overall delay is sum of delays
Overall occupancy (1/bandwidth) is biggest of occupancies
Comm Cost = frequency * (Comm time - overlap)
General model for data transfer: applies to cache misses too
Fundamental Design Issues

68. Summary of Design Issues
Functional and performance issues apply at all layers
Functional: Naming, operations and ordering
Performance: Organization, latency, bandwidth, overhead, occupancy
Replication and communication are deeply related
Management depends on naming model
Goal of architects: design against frequency and type of operations that occur at communication abstraction, constrained by tradeoffs from above or below
Hardware/software tradeoffs
Fundamental Design Issues

69. Recap
Parallel architecture is important thread in evolution of architecture
At all levels
Multiple processor level now in mainstream of computing
Exotic designs have contributed much, but given way to convergence
Push of technology, cost and application performance
Basic processor-memory architecture is the same
Key architectural issue is in communication architecture
How communication is integrated into memory and I/O system on node
Fundamental design issues
Functional: naming, operations, ordering
Performance: organization, replication, performance characteristics
Design decisions driven by workload-driven evaluation
Integral part of the engineering focus
Fundamental Design Issues