1. Who We Are? Detector Building Group of KFKI-RMKI (Research Institute for Particle and Nuclear Physics), Budapest, HUNGARY

2. Our Results Previous projects:  Designing Fibre Channel test equipments (1994-1998)  Portable, 266 Mb/s Fibre Channel Tester  Fibre Channel Preprocessor for Logic Analyzers  Designing high-speed data transmission interfaces for CERN detectors (1996 -...)  Protocol design and verification  FC, GbE, physical layer components  Hardware design  Software: linux drivers, test programs, program library

3. S-LINK (CERN)  S-LINK interface cards for CERN (mostly used at ATLAS)  simple, unidirectional link interface  first successful version was designed by KFKI-RMKI  newer versions, now: 2.5 Gb/s  Also used at MPI, Garching and at several HEP and other scientific institutes all over the world

4. ALICE Detector Data Link (DDL)

5. DDL  Detector Data Link (DDL) for the ALICE detector at CERN  2.5 Gb/s, duplex link  advanced featues  Test devices for DDL (Front-end emulator, DDL link emulator, etc.)  PCI Read-Out Receiver Card (RORC) for DDL  1st version: 33 MHz, 32-bit PCI card  2nd version: 66 MHz, 64-bit PCI with 2 integrated DDL interfaces

6. Front-end Electronics Front-end Electronics DDL Source Interface Unit (SIU) DDL Source Interface Unit (SIU) TAPTAP DDL Destination Interface Unit (DIU) DDL Destination Interface Unit (DIU) Read-out Receiver Card (RORC) 66 MHz 64-bit PCI Read-out Receiver Card (RORC) 66 MHz 64-bit PCI Front-end Bus 32 bit JTAG BST lines 32 bit 4 1 FEE - SIU Interface DIU -RORC Interface optical cable, max 350 m DDL DDL Interfaces

7. DDL Features Interface:  Full duplex 32-bit data path on the destination interface (DIU card)  Half duplex 32-bit data path on the source interface (SIU card)  Full duplex flow control (XON/XOFF)  Interface clock up to 66 MHz (easy integration with PCI 66)  264 MB/s peak data rate, 240 MB/s sustained bandwidth (max.) Implementation:  Duplex LC optical link up to 300 m  2x FC or 2x GbE physical layer components  Small Form Factor Pluggable (SFP) optical transceivers  Bit error rate < 10 -12  Robust error detection: very low undetected bit error rate < 10 -40  Automatic link synchronization and management Extras:  Stand-by support (low power consumption)  In-system reconfiguration / Remote system upgrade  Monitoring of the aging of laser diode of optical transceivers  JTAG Boundary Scan Test interface for the Front-End electronics

8. Outlook to PCI Express PCI Express (formerly 3GIO) a „third generation” high performance I/O bus (1st generation: ISA, EISA, VESA, 2nd generation: PCI, PCI-X) PCI evolution (PCI, PCI-X, PCI-Express) PCI Express is software compatible to PCI and PCI-X

9. PCI Evolution  PCI  „multi-drop” parallel bus  conventional PCI: 33 MHz, 32-bit, max 4-5 card slots per bus  newer versions: 66 MHz, 64-bit, 1 (max 2) card slots per bus (!)  PCI-X  parallel bus, backward compatible (hw and sw)  66 MHz, 32/64-bit, max 4-5 card slots per bus  133 MHz, 32/64-bit,max 1-2 card slots per bus  266, 532 MHz versions: max 1 card slot per bus (!) Buses can be bridged to each other (complex, expensive)  PCI-XP  It is still a local /IO bus, a „PCI bus”, but the connections between devices are serial, point-to-point interconnections  devices are interconnected via switch(es)  large number of devices can be interconnected  highly scalable, hot-plug, hot-swap, QoS, etc.

10. Limitations of PCI and PCI-X  Conventional 33 MHz PCI system  low bandwith to nowaday’s needs  66 MHz, 133 MHz (PCI-X),...  only few devices can be interconnected (only 1 or two) on a single bus because of the strict electrical load and timing constraints  Further limitations of PCI architecture  inefficient solutions in:  data transfer cycles (wait states)  accessing of system memory  interrupt handling  error handling

11. PCI Express  However, the basic problems of a parallel bus system (electrical load and timing constraints, lack of hot- pluggability, lack of scalability, etc.) can be solved only by a complete redesign of the architecture.  This resulted in PCI Express.

12.  Serial point-to-point interconnect between two devices  1x, 2x, 4x, 8x, 12x, 16x, 32x type links  1, 2, 4,... 32 bidirectional signal pairs (lanes)  1 lane: 2.5 Gb/s now, up to 10 Gb/s later  Low voltage, differential signaling (LVDS)  AC coupled  Data is encoded: 8B/10B The PCI Express Link Device A Device B (e.g. a switch) a PCI Express link 1 to 32 lanes

13. Bandwith  Scalable: more lanes /link: higher bandwith  2.5 Gb/s per lane, 8B/10B encoding  Simultaneous traffic in both directions  1x type link  500 MByte/s aggregate bandwith  250 MByte/s per direction  32x type link:  16 GByte/s aggregate bandwith  (8 GByte/s per direction)

14. Backward compatibility SSSSupports familiar PCI transactions such as memory read/write, IO read/write and configuration read/write SSSSame memory, IO and configuration address space as in PCI and PCI-X EEEExisting OSs and driver software will run in a PCI Express system without any modifications SSSSupports chip-to-chip interconnect and board-to-board interconnect via cards and connectors similar to the present PCI systems PPPPCI Express motherboard will have a similar form factor to existing ATX motherboards in PCs

15. Improvements, Benefits  Fast, highly scalable, serial point-to-point interconnect between two devices  bytes striped accross the lanes  more lanes per link: faster transmission  Packet-based communication protocol  Packets are transmitted serially  CRC embedded in each packet (auto retry: link level error correction)  Buffer-to buffer (link level) flow control  no need the packet retry  Message Signaled Interrupt (MSI) architecture  No side-band signals  hot-plug, error handling, interrupt signaling and else are accopmlished in-band  Multiple devices are interconnected via switches  Large number of devices can be connected together in a system  Much fewer pins per device package  Reduces chip and board design cost and design complexity  Reconfigurable, hot-plug, hot-swap, improved power management, etc.  Quality of Service (QoS) features: Traffic Classes, Virtual Channels  Configuration address space of devices is extended from 256B to 4KB  This needs new software

16. Outside the box?  Initial focus of usage: inside the box Add-in card switch PCI-XP links  It is expected that later it will also be used outside the box for I/O expansion Add-in card switch PCI-XP links connector optical cable PCI-XP link

17. Mechanical Form Factors  Connector and daughter card form factors are under specification  The main type of connectors is very similar to the present PCI card edge connector  A 1x type card can be inserted in a x4 type slot, a.s.o.  Like with PCI, there will be other form factors  Server I/O module  Mini PCI Express card and connector (e.g. for notebooks)  mezzanine type card  NEWCARD (will replace CardBus PC card), etc.

18. Compete or complete?  PCI Express will coexist with PCI / PCI-X in the same system  PCI Express intends to replace AGP (graphics card if), but will not replace Serial ATA (hard disk, CD, etc)  HyperTransport  onboard chip-to-chip interconnect  said to be complementary to PCI Express  RapidIO  as an onboard chip-to-chip interconnect: complementary to PCI-Express  as a system interconnect: a competitive technology  1394b, USB2.0, Fibre Channel, Gb Ethernet, Infinband, etc.  PCI Express can be a system interconnect that bridges these technologies  Will it be?