Architectural Freedom with Serial Interconnects NDIA Conference San Diego, CA October 20-23, 2003 Emmanuel Eriksson – Dy 4 Systems

Interconnect Trends Bus vs. Switched Device 1980s Device Bridge 1990s 2000s Device Switched Interconnect

Shortcomings of Parallel Buses scale number of connections scale physically scale bandwidth support multiple traffic types support high availability You only need to encounter one limitation to make the move to a new architecture Buses Dont:

Typical Embedded Distributed System Requirements Large amount of data movement Large amount of data processing Large number of processing elements Often multiple processor types –DSP, CPU and NPU Wide range of I/O devices –Sensors –Imaging and Video devices –Storage –Network Interfaces –Displays

Bus Based Architecture CPUCPU IOCARDIOCARD IOCARDIOCARD IOCARDIOCARD IOCARDIOCARD IOCARDIOCARD IOCARDIOCARD IOCARDIOCARD PCI Bus CPU Bus Bridge Memory SCSI Ethernet USB Sensor Interface Bus Bridge Single address map Limited to a single processor or single bus owner at a time Difficulty scaling processing requirements with I/O needs

LAN Based Architecture Internet Protocol is a sophisticated protocol that is aimed at a complex problem: Interconnect hundreds of thousands of computers –Handle extreme latencies –Handle lost packets –Handle corrupted packets –Handle out-of-order packet arrival

Limitations of Current Architectures Difficulty balancing computational power with I/O requirements –No support for both processor-to-processor and processor-to-device communication (Ethernet and bus architectures can do either but not both) Lossy LAN transport do not meet lossless data collection requirements without additional software –Ethernet will drop packets if congestion occurs High latency or limited bandwidth scaling –Moving a 64 bytes packet takes > 100 microseconds with Ethernet versus a couple of microseconds with a switch fabric Limited Class of Service capability –Ethernet has no separate queuing or buffering for CoS –Buses dont have any CoS

Serial Switched Interconnects Scalability, Density, Flexibility Supports disaggregated and virtual systems with highly flexible upgradeability High Availability & Reliability Failures detected, reported & managed. Lossless fabric with CRC protection. One failure cant take down system Low Cost Leverages PCI infrastructure and a wealth of products with lean, protocol-agnostic switching. High Performance Unified Fabric Multiple types and classes of traffic supported on low latency, high bandwidth interconnect. Device CPU/Device Device CPU/Device Device CPU/Device Switched Interconnect

StarFabric & PCI Express Advanced Switching (AS) Scalable, Extensible Serial Switched Interconnects –Lean, High Performance, Low Cost Switching Architectures –Layered Approach Promotes Efficient Modularity StarFabric –Open standard switched interconnect (StarFabric Trade Association) –Based on 622Mb/s SERDES technology 4X data bandwidth of 2 Gb/s Links can extend beyond single chassis up to 10 meters through standard CAT5 cabling –In production since Spring 2002 PCI Express - Advanced Switching (AS) –Based on the PCI Express architecture –Based on 2.5Gb/s SERDES technology 4X data bandwidth of 8 Gb/s –Available in late 2004

Multi-Layered Protocols Software Physical Point to point, serial, differential, hot-plug, inter-op form factors Data Integrity Data Link 622Mb/s for StarFabric 2.5Gb/s for AS Transaction StarFabric & Advanced Switching Packet Switching Hardware PCI Bus Driver (init, enum, config) P-to-P Driver SF/AS Driver Operating System Legacy Peer-Peer CRC, Link Synchronization, Auto negotiation, Link status Data routing, Credit based flow control, Event and error handling, etc.

Example Interconnect: StarFabric Bus Bridge Bus With one Switch device, connect 6 Bus segments Switch Redundant fabric interconnect accomplished with a 2nd Switch device Switch StarFabric Frames at 2.5G Transmit and 2.5G Receive Parallel Bus PCI or TDM or Proprietary

New Degrees of Architectural Freedom SF or AS Fabric Network Gateway Processor Boards Solid State Storage Storage Blades SCSI/FC Cntrl SCSI/FC Cntrl Bridge SCSI/FC Cntrl Bridge Processor Chipset Memory Bridge CPU Chipset Memory Bridge GigE MAC Bridge DSP Memory Bridge DSP Memory Bridge DSP DSP Boards I/O uC I/O Bridge uC I/O uC I/O Bridge uC Other IO: Sensors EO/IR Video Voice comms Etc... Switched Interconnect

I/OCARDI/OCARD CPUCPU SWITCHSWITCH I/OCARDI/OCARD SWITCHSWITCH STORAGESTORAGE STORAGESTORAGE CPUCPU 1GLAN1GLAN CPUCPU SWITCHSWITCH CPUCPU I/OCARDI/OCARD SWITCHSWITCH STORAGESTORAGE 1GLAN1GLAN STORAGESTORAGE CPUCPU CPUCPU SWITCHSWITCH CPUCPU SWITCHSWITCH STORAGE2STORAGE2 STORAGE2STORAGE2 10 G L A N CPU2CPU2 CPU2CPU2 CPU2CPU2 CPU2CPU2 DSPDSP DSPDSP DSPDSP DSPDSP DSPDSP DSPDSP DSPDSP DSPDSP I/OCARDI/OCARD I/OCARDI/OCARD Adaptive Infrastructure Life Cycle Benefits Initial PurchaseInitial Upgrade Scale capacity Mid Life Kicker Upgrade CPUs & Storage Increase LAN speeds

Radar Imaging System I/O from RF Modules DSP farm to process image Centralized computer and image displays CPU Chipset Memory Bridge DSP Memory Bridge DSP Storage Blades SCSI/FC Cntrl Bridge I/O uC RF TX/Rx Bridge uC DSP Memory Bridge DSP Memory Bridge DSP I/O uC RF TX/Rx Bridge uC I/O uC RF TX/Rx Bridge uC Switched Interconnect

Dy 4 System Solutions StarFabric Interconnect & Interprocessor Communication Software CHAMP-AVII (Quad PowerPC) CHAMP-FX (Dual FPGA Board) StarReach (PMC Carrier Board) DMV-182 (Single Board Computer)

Advantages of Distributed Switching Separate Switch Card Requires extra slot Single point of failure Extra LRU item logistics Switch card X Backplane switching Four slots Active circuits on backplane Switch circuit on backplane X StarFabric Passive Backplane X X Switching on card No active backplane Four slots used

Summary Switched Interconnects like StarFabric and PCI Express AS: –Provide low cost, high performance interconnects for multislot, distributed architectures –Leverage the PCI/PCI-X infrastructure and volumes –Provide CPU-to-CPU and CPU-to-device communication –Reduces processor overhead and system latencies –Provides a reliable interconnect for multiple classes of service –Enables design flexibility and system disaggregation –Allows flexible fabric architecture, centralized or distributed switching

Thank You! Contact information for follow up questions or discussions... Emmanuel Eriksson Technology Architect Technology Innovation Group Dy 4 Systems Inc. (613) x5462 or visit