CDA-5155 Computer Architecture Principles Fall 2000 Interconnection Networks

Review I/O gap Storage I/O performance measures Processor interface issues A Little queuing theory I/O benchmarks Redundant arrays of inexpensive disks (RAID) UNIX file systems Storage architectures

Overview Architectural implications of networks Performance issues HW & SW overhead Interconnect latency, bisection BW Media Cost, distance Shared vs. Switched Media determines BW HW and SW Interface to computer Overhead, latency, bandwidth Connecting multiple computers

I/O to Other Computers Processor Cache Memory - I/O Bus Main I/O I/O interrupts Processor Cache Memory - I/O Bus Main I/O I/O I/O Memory Controller Controller Controller Graphics Disk Disk Network IDEAL: high bandwidth low latency

Networks Collection of computers as one big computer Technical issues Direct (point-to-point) vs. broadcast Topology (e.g., bus, ring, tree) Standards and protocols Internetworking, Routing algorithms, Switching Wiring (e.g., media: copper, coax, fiber) Performance issues: Latency Bandwidth Cost Reliability

Examples of Major Networks

Types of Interconnects MPP networks (SP2): 100s nodes; 25 meters Local Area Networks (Ethernet): 100s nodes; 1 Km Wide Area Network (ATM): 1000s nodes; 5,000 Km

XAN’s SAN - system area network (MPP network) LAN - local area network Connects (100’s – 1000’s) homogeneous nodes Physical extent is small – (much) less than 25m Main focus is high bandwidth and low latency Supported by MPP industry - very model specific LAN - local area network General connectivity, 100’s of heterogeneous hosts Physical extent: a few kilometers Performance is typically mundane Supported by workstation industry - open system model WAN - wide area network General connectivity, 1000’s of heterogeneous nodes High bandwidth but latency is usually horrible Physical extent: 1000’s of kilometers Supported by the telecom industry - open system model

A Simple Network Remote memory read (load/store) Send bits between 2 computers Queue (FIFO) on each end, Full duplex Protocol (rules for communication) Packet format 0/1 Payload (32 bits) Request/Response Address / Data

Communication Complexity More than 2 computers want to communicate Need computer address fields in packet, routing Transmission errors Add error detection field in packet (e.g., CRC) Packet loss NAK, time outs Multiple processes/machine Queue per process to provide protection Different capacities Flow control algorithms => More complex protocols and packet formats

Send and Receive Protocols Send protocol Application calls OS to send data OS copies data to OS buffer OS calculates checksum, starts timer OS sends data to network interface HW and says start Receive protocol 3. OS copies data from NIC to OS buffer 2. OS calculates checksum, if matches send ACK; if not, deletes message (sender resends when timer expires) 1. If OK, OS copies data to user address space and signals application to continue

Network Latency vs. Node Overhead

Network Performance Metrics Sender Overhead Transmission time (size ÷ bandwidth) Sender (processor busy) Time of Flight Transmission time (size ÷ bandwidth) Receiver Overhead Receiver (processor busy) Transport Latency Total Latency Total Latency = Sender Overhead + Time of Flight + Message Size ÷ BW + Receiver Overhead Includes header/trailer in BW calculation?

Example Interconnect MPP LAN WAN Example CM-5 Ethernet ATM Bisection BW N*5MB/s 1.125 MB/s N*10 MB/s Int./Link BW 20 MB/s 1.125 MB/s 10 MB/s Transport Latency 5 µsec 15 µsec 50 to 10,000 µs HW Overhead to/from 0.5/0.5 µs 6/6 µs 6/6 µs SW Overhead to/from 1.6/12.4 µs 200/241 µs 207/360 µs TCP/IP on LAN/MAN Software overhead dominates in LAN/WAN

Latency Example Parameters 230 usec sender overhead, 270 usec receive overhead 10 Mbit bandwidth (assume no contention) 1000 byte message (fits in a single packet) Ignore size of trailer and header transmissivity = .5 (very good optical fiber or coax) No repeater delay (not realistic for the 1000 km case) Latency0.1km = 230 + 0.1km / (0.5 * 299,792.5) + 1000*8/10 + 270 = 230 + 0.67+ 800 + 270 = 1301 µsec Latency1000km = 230 + 1000 km / (0.5*299,792.5)+ 1000*8/10+ 270 = 230 + 6671+ 800 + 270 = 7971 µsec Long time of flight => complex WAN protocol

Overhead, Bandwidth, Size

NFS Message Sizes 95% Msgs, 30% bytes for packets < 200 bytes 100% 90% 80% Msgs 70% 60% Bytes 50% Cumulative % 40% 30% 20% 10% Packet size 0% 1024 2048 3072 4096 5120 6144 7168 8192 95% Msgs, 30% bytes for packets < 200 bytes >50% data transferred in packets = 8KB

Overhead and Delivered BW 1000.00 1 100.00 10 Delivered BW (MB/sec) 10.00 100 1000 1.00 0.10 MinTime 1 10 100 1000 one-way Peak BW (MB/sec) µsecs BW model: Time = overhead + msg size/peak BW >50% data transferred in packets = 8K

Hardware Interface Where to connect network to computer? Cache consistent to avoid flushes? (=> memory bus) Latency and bandwidth? (=> memory bus) Standard interface card? (=> I/O bus) MPP => memory bus; LAN, WAN => I/O bus CPU Network Network $ IDEAL: High bandwidth Low latency Standard interface I/O Controller I/O Controller L2 $ Memory Bus I/O bus Memory Bus Adaptor

Software Interface How to connect network to software? Things to avoid Programmed I/O: low latency DMA: best for large messages Receiver interrupted or received polls? Things to avoid Invoking operating system in common case Operating at uncached memory speed (e.g., check status of network interface) Copying overhead

CM-5 Software Interface CM-5 (MPP) Time per poll 1.6 µsecs; time per interrupt 19 µsecs Minimum time to handle message: 0.5 µsecs Enable/disable 4.9/3.8 µsecs Polling or interrupt? Solution: Always enable interrupts, have interrupt routine poll until no messages pending Low rate => interrupt High rate => polling Overhead (µsecs) 100 90 80 70 60 Polling 50 40 30 Interrupts 20 10 10 20 30 40 50 60 70 80 90 100 message interarrival time (µsecs)

Network Media Copper, 1mm think, twisted to avoid Twisted Pair: Copper, 1mm think, twisted to avoid antenna effect (telephone) Air (RF): Coaxial Cable: Used by cable companies: high BW, good noise immunity Plastic Covering Braided outer conductor Insulator Copper core Light: 3 parts are cable, light source, light detector. Multimode light disperse (LED), Single mode single wave (laser) Fiber Optics Total internal Transmitter Air reflection Receiver – L.E.D – Photodiode – Laser Diode light source Silica

Connecting Multiple Computers Shared Media vs. Switched: “point-to-point” connections Aggregate BW in switched network is many times shared point-to-point faster since no arbitration, simpler interface Arbitration in Shared networks Central arbiter for LAN? Listen to check if being used (“Carrier Sensing”) Listen to check if collision (“Collision Detection”) Random resend to avoid repeated collisions; not fair arbitration; OK if low utilization