XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Grzegorz Jablonski, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 Communication in ATCA

Grzegorz Jablonski, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 2 Agenda Communication interfaces in ATCA standard Latency requirements in different parts of the system Latency in selected communication standards Low latency links Proposed ATCA PCIe communication hierarchy Possible hardware implementation solutions

Grzegorz Jablonski, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 3 Communication interfaces in ATCA standard PICMG3.1 Ethernet / Fibre Channel PICMG3.2 Infiniband PICMG3.3 PCI Express / Advanced Switching PICMG3.4 Starfabric PICMG3.5 RapidIO PICMG3.6 PRS fabric

Grzegorz Jablonski, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 4 Latency requirements in different parts of the system cryomodule klystron 8 channel board 3x 8 channel AMC board 3x 8 channel board 3x 8 channel AMC board 3x 8 channel board 3x 8 channel AMC board 3x 8 channel AMC board 3x Carrier board DAC AMC board Control System 24 channels high bandwidth links, latency not important, ~1Gb/s low latency links, critical for on-line operation, 36 bit x 5MHz online computation Carrier board low protocol overhead links implemented using RocketIO

Grzegorz Jablonski, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 5 Latency requirements in different parts of the system Total overall latency of the following links: - low latency links - low protocol overhead links should not exceed ~800 ns

Grzegorz Jablonski, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 6 Latency in selected communication standards Rocket IO –direct peer-to-peer connections –latency around 300 ns (Virtex 2 Pro) and 100 ns (Virtex 5)‏ PCI Express –connections via switch –minimum latency ns depending on payload size –up to 10 us with higher switch load Gigabit Ethernet –2.5 us 10 Gigabit Ethernet – ns

Grzegorz Jablonski, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 7 Low latency links Low latency links are the links used for on-line communication amongst FPGA chips. They provide sufficient bandwidth and guarantee constant latency of the transmission below 50 ns. The will be implemented using 7 differential data lines and one clock line in LVDS standard For the communication amongst the chips on the separate carrier boards, special low protocol overhead links will be used. They will be implemented using RocketIO, which provides means of synchronization of the transmission and is immune to skew (the transmission clock is recovered from data stream).

Grzegorz Jablonski, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 8 Proposed ATCA PCIe communication hierarchy Root Complex implemented in ColdFire PowerPC Virtex 5 contains hardware PCIe endpoint block, with a free licence

Grzegorz Jablonski, Technical University of Lodz, Department of Microelectronics and Computer Science XFEL-LLRF-ATCA Meeting, 3-4 December 2007 XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser 9 Possible hardware implementation solutions Two alternatives for implementing a root complex: Embedded processor with built-in root complex PCIe core in an FPGA The root complex FPGA IP block available from PLDA ( Problems getting an evaluation license Problem getting a price quotation Proposed solution – use MPC85xx Freescale PowerQUICC III Processor