1. CDA-5155 Computer Architecture Principles Fall 2000
Interconnection Networks

2. Review I/O gap Storage I/O performance measures
Processor interface issues
A Little queuing theory
I/O benchmarks
Redundant arrays of inexpensive disks (RAID)
UNIX file systems
Storage architectures

3. Overview Architectural implications of networks Performance issues
HW & SW overhead
Interconnect latency, bisection BW
Media
Cost, distance
Shared vs. Switched Media determines BW
HW and SW Interface to computer
Overhead, latency, bandwidth
Connecting multiple computers

4. I/O to Other Computers Processor Cache Memory - I/O Bus Main I/O I/O
interrupts
Processor
Cache
Memory - I/O Bus
Main
I/O
I/O
I/O
Memory
Controller
Controller
Controller
Graphics
Disk
Disk
Network
IDEAL: high bandwidth
low latency

5. Networks Collection of computers as one big computer Technical issues
Direct (point-to-point) vs. broadcast
Topology (e.g., bus, ring, tree)
Standards and protocols
Internetworking, Routing algorithms, Switching
Wiring (e.g., media: copper, coax, fiber)
Performance issues:
Latency
Bandwidth
Cost
Reliability

6. Examples of Major Networks

7. Types of Interconnects
MPP networks (SP2): 100s nodes; 25 meters
Local Area Networks (Ethernet): 100s nodes; 1 Km
Wide Area Network (ATM): 1000s nodes; 5,000 Km

8. XAN’s SAN - system area network (MPP network) LAN - local area network
Connects (100’s – 1000’s) homogeneous nodes
Physical extent is small – (much) less than 25m
Main focus is high bandwidth and low latency
Supported by MPP industry - very model specific
LAN - local area network
General connectivity, 100’s of heterogeneous hosts
Physical extent: a few kilometers
Performance is typically mundane
Supported by workstation industry - open system model
WAN - wide area network
General connectivity, 1000’s of heterogeneous nodes
High bandwidth but latency is usually horrible
Physical extent: 1000’s of kilometers
Supported by the telecom industry - open system model

9. A Simple Network Remote memory read (load/store)
Send bits between 2 computers
Queue (FIFO) on each end, Full duplex
Protocol (rules for communication)
Packet format
0/ Payload (32 bits)
Request/Response
Address / Data

10. Communication Complexity
More than 2 computers want to communicate
Need computer address fields in packet, routing
Transmission errors
Add error detection field in packet (e.g., CRC)
Packet loss
NAK, time outs
Multiple processes/machine
Queue per process to provide protection
Different capacities
Flow control algorithms
=> More complex protocols and packet formats

11. Send and Receive Protocols
Send protocol
Application calls OS to send data
OS copies data to OS buffer
OS calculates checksum, starts timer
OS sends data to network interface HW and says start
Receive protocol
3. OS copies data from NIC to OS buffer
2. OS calculates checksum, if matches send ACK; if not, deletes message (sender resends when timer expires)
1. If OK, OS copies data to user address space and signals application to continue

12. Network Latency vs. Node Overhead

13. Network Performance Metrics
Sender
Overhead
Transmission time
(size ÷ bandwidth)
Sender
(processor
busy)
Time of
Flight
Transmission time
(size ÷ bandwidth)
Receiver
Overhead
Receiver
(processor
busy)
Transport Latency
Total Latency
Total Latency = Sender Overhead + Time of Flight +
Message Size ÷ BW + Receiver Overhead
Includes header/trailer in BW calculation?

14. Example Interconnect MPP LAN WAN Example CM-5 Ethernet ATM
Bisection BW N*5MB/s MB/s N*10 MB/s
Int./Link BW 20 MB/s MB/s 10 MB/s
Transport Latency 5 µsec µsec 50 to 10,000 µs
HW Overhead to/from 0.5/0.5 µs 6/6 µs 6/6 µs
SW Overhead to/from 1.6/12.4 µs 200/241 µs 207/360 µs
TCP/IP on LAN/MAN
Software overhead dominates in LAN/WAN

15. Latency Example Parameters
230 usec sender overhead, 270 usec receive overhead
10 Mbit bandwidth (assume no contention)
1000 byte message (fits in a single packet)
Ignore size of trailer and header
transmissivity = .5 (very good optical fiber or coax)
No repeater delay (not realistic for the 1000 km case)
Latency0.1km = km / (0.5 * 299,792.5) *8/ = = 1301 µsec
Latency1000km = km / (0.5*299,792.5)+ 1000*8/
= = 7971 µsec
Long time of flight => complex WAN protocol

16. Overhead, Bandwidth, Size

17. NFS Message Sizes 95% Msgs, 30% bytes for packets < 200 bytes
100%
90%
80%
Msgs
70%
60%
Bytes
50%
Cumulative %
40%
30%
20%
10%
Packet size 0%
1024
2048
3072
4096
5120
6144
7168
8192
95% Msgs, 30% bytes for packets < 200 bytes
>50% data transferred in packets = 8KB

18. Overhead and Delivered BW 1
100.00 10
Delivered BW
(MB/sec)
10.00
100
1000
1.00
0.10
MinTime 1 10
100
1000
one-way
Peak BW (MB/sec)
µsecs
BW model: Time = overhead + msg size/peak BW
>50% data transferred in packets = 8K

19. Hardware Interface Where to connect network to computer?
Cache consistent to avoid flushes? (=> memory bus)
Latency and bandwidth? (=> memory bus)
Standard interface card? (=> I/O bus)
MPP => memory bus; LAN, WAN => I/O bus
CPU
Network
Network $
IDEAL:
High bandwidth
Low latency
Standard interface
I/O
Controller
I/O
Controller
L2 $
Memory Bus
I/O bus
Memory
Bus Adaptor

20. Software Interface How to connect network to software? Things to avoid
Programmed I/O: low latency
DMA: best for large messages
Receiver interrupted or received polls?
Things to avoid
Invoking operating system in common case
Operating at uncached memory speed (e.g., check status of network interface)
Copying overhead

21. CM-5 Software Interface
CM-5 (MPP)
Time per poll 1.6 µsecs; time per interrupt 19 µsecs
Minimum time to handle message: 0.5 µsecs
Enable/disable 4.9/3.8 µsecs
Polling or interrupt?
Solution: Always enable interrupts, have interrupt routine poll until no messages pending
Low rate => interrupt
High rate => polling
Overhead (µsecs)
100 90 80 70 60
Polling 50 40 30
Interrupts 20 10 10 20 30 40 50 60 70 80 90
100
message interarrival time (µsecs)

22. Network Media Copper, 1mm think, twisted to avoid
Twisted Pair:
Copper, 1mm think, twisted to avoid
antenna effect (telephone)
Air (RF):
Coaxial Cable:
Used by cable companies: high BW, good noise immunity
Plastic Covering
Braided outer conductor
Insulator
Copper core
Light: 3 parts are cable, light source, light detector.
Multimode light disperse (LED), Single mode single wave (laser)
Fiber Optics
Total internal
Transmitter
Air
reflection
Receiver
– L.E.D
– Photodiode
– Laser Diode
light
source
Silica

23. Connecting Multiple Computers
Shared Media vs. Switched: “point-to-point” connections
Aggregate BW in switched network is many times shared
point-to-point faster since no arbitration, simpler interface
Arbitration in Shared networks
Central arbiter for LAN?
Listen to check if being used (“Carrier Sensing”)
Listen to check if collision (“Collision Detection”)
Random resend to avoid repeated collisions; not fair arbitration;
OK if low utilization