1. The TrainBuilder ATCA Data Acquisition Board for the European XFEL The TrainBuilder ATCA Data Acquisition Board for the European-XFEL John Coughlan, Chris Day, James Edwards, Ed Freeman, Senerath Galagedera and Rob Halsall Science & Technology Facilities Council Rutherford Appleton Laboratory Oxfordshire, United Kingdom E-mail: john.coughlan@stfc.ac.uk Presented by John Coughlan. TWEPP Oxford. September 2012.

2. The TrainBuilder ATCA Data Acquisition Board for the European XFEL 2 European-XFEL DAQ TrainBuilder demonstrator board FPGA Firmware System Applications Presented by John Coughlan. TWEPP Oxford. September 2012. Topics

3. The TrainBuilder ATCA Data Acquisition Board for the European XFEL XFEL DAQ Data Flow 3 Large 2D Pixel Detectors 1M Pixel, 2B/Pixel, 512 Images @10Hz (X-Ray pulse Train) 10 Gbyte per second per Megapixel rate 10G Ethernet using UDP protocol Modularity : 16 x 10G Links off detector per Megapixel Scalable Full Data Rate to PC Layer 3 Train Builder Detector 2D Camera 10G Switch10G servers PC Layer 10G Common Systems XFEL DAQ Presented by John Coughlan. TWEPP Oxford. September 2012.

4. The TrainBuilder ATCA Data Acquisition Board for the European XFEL Image Train Building 4 Presented by John Coughlan. TWEPP Oxford. September 2012. FEMs Train Builder FEMs PC SWITCH PC SWITCH PC Train Builder N x 512 10Gbps Train Builder Detector 2D Camera 10G servers PC Layer Partial Images 128 KB Full Images 2 MB Detector Specific Common XFEL DAQ N= 16 Deep Buffers Switch 10G Links

5. The TrainBuilder ATCA Data Acquisition Board for the European XFEL TrainBuilder Demonstrator 5 Presented by John Coughlan. TWEPP Oxford. September 2012. Train Builder Demonstrator ATCA form factor 5 x Xilinx V5FXT100 FPGAs Analogue X-point 80x80 4 x FMCs dual SFP+ 10Gb 8 x 2GB DDR2 SODIMM VLP Modules 4 x QDRII SRAMs Links Configured as Input or Output by software Connects to either Detectors or PC Farm

6. The TrainBuilder ATCA Data Acquisition Board for the European XFEL TrainBuilder FE x 4 6 Presented by John Coughlan. TWEPP Oxford. September 2012. Virtex5 FX100T, Dual PPC DESY FMC 2 x 10Gbps SFP+ 2 x DDR2 VLP SODIMMs 2 GByte QDRII SRAM 8 MByte SRAM PPC code

7. The TrainBuilder ATCA Data Acquisition Board for the European XFEL TrainBuilder FE x 4 7 Presented by John Coughlan. TWEPP Oxford. September 2012. Virtex5 FX100T, Dual PPC DESY FMC 2 x 10Gbps SFP+ QDRII SRAM 8 MByte QDRII SRAM Pixel reordering DDR2 Partial Images train buffer To Xpoint 10G Inputs from Detector Configured as INPUT : Data from Detectors 2 x DDR2 VLP SODIMMs 2 GByte

8. The TrainBuilder ATCA Data Acquisition Board for the European XFEL TrainBuilder FE x 4 8 Presented by John Coughlan. TWEPP Oxford. September 2012. Virtex5 FX100T, Dual PPC DESY FMC 2 x 10Gbps SFP+ QDRII SRAM 8 MByte 10G Outputs to Farm QDRII SRAM Image Tiling DDR2 Full Images train buffer From Xpoint Configured as OUTPUT : Data to PC Farm 2 x DDR2 VLP SODIMMs 2 GByte

9. The TrainBuilder ATCA Data Acquisition Board for the European XFEL TrainBuilder 10G link FMC 9 Presented by John Coughlan. TWEPP Oxford. September 2012. FPGA Mezzanine Card FMC ANSI/VITA 57 Developed by DESY Electronics Group Dual 10 Gbps SFP+ Vitesse 10G PHYs XAUI 4 x 3.125 Gbps

10. The TrainBuilder ATCA Data Acquisition Board for the European XFEL TrainBuilder Demonstrator 10 Presented by John Coughlan. TWEPP Oxford. September 2012. Master FPGA Boot MMC Spartan3AN FPGA Xpoint Crosspoint Switch Mindspeed 80x80 6.5 Gbps 64 Tx & Rx @ 2.5 Gbps Switch changes @ ~ 10 Hz Master FPGA Virtex5 FXT100 Manages data switching PPC GbE TCP/IP server Python scripts on client PC FPGA Configuration CF card Boot Spartan3 with FLASH GbE Ctrl GbE Ctrl RJ45

11. The TrainBuilder ATCA Data Acquisition Board for the European XFEL TrainBuilder ATCA Interfaces 11 Presented by John Coughlan. TWEPP Oxford. September 2012. ATCA Zone 3 Cable/Rear Transition Module ATCA Zone 2 GbE Ctrl PCIe ATCA Zone 1 JTAG Zone 3 Custom RTM 32 Tx & Rx data lanes to Crosspoint + clock&controls Zone 2 Std interfaces Zone 1 Power 48V MMC 3V3 ATCA 48V DC-DC

12. The TrainBuilder ATCA Data Acquisition Board for the European XFEL TrainBuilder Data Rates 12 Presented by John Coughlan. TWEPP Oxford. September 2012. 10G XAUI 4 x 3.125 Gbps > 1 GB/sec Crosspoint Aurora 4 lanes 64B/66B @ 2.5 Gbps > 1 GB/sec DDR2 (1 per link) 2 GB Dual PPC Hard IP Embedded ~ 1 GB/sec Read & Write (Long bursts) QDRII SRAM (1 per 2 links) 8 MB 32 bit R & W @ 250 MHz ~ 1 GB/sec per link (2 word burst) Detector (512 bunches/train) ~ 640 MB/sec per 10G LINK Total B/W per TB board 8 Links ~ 8 GB/sec

13. The TrainBuilder ATCA Data Acquisition Board for the European XFEL 13 Firmware Presented by John Coughlan. TWEPP Oxford. September 2012.

14. The TrainBuilder ATCA Data Acquisition Board for the European XFEL FE FPGA Firmware 14 Presented by John Coughlan. TWEPP Oxford. September 2012. Aurora Core 0 XPOINT 4 TX PAIRS 4 RX PAIRS 4 TX PAIRS 4 RX PAIRS Aurora Core 1 XAUI Core 0 XAUI Core 1 Power PC Memory Controller Subsystem 4 TX PAIRS 4 RX PAIRS 4 TX PAIRS 4 RX PAIRS XTAL PLL XTAL PLL DUAL 10G PHY FMC PHY SFP+ PHY SFP+ TB FPGA 10G UDP Core 1 Power PC Memory Controller Subsystem 10G UDP Core 0 ‘XCTRL’ DDR2 SDRAM Image Buffers Storage DDR2 SDRAM QDRII SRAM Image Manipulation QDRII SRAM Aurora 64B/66B XAUI QDRII Interface QDRII Interface LOCAL LINK Protocol

15. The TrainBuilder ATCA Data Acquisition Board for the European XFEL FE FPGA Firmware 15 Presented by John Coughlan. TWEPP Oxford. September 2012. Aurora Core 0 XPOINT 4 TX PAIRS 4 RX PAIRS 4 TX PAIRS 4 RX PAIRS Aurora Core 1 XAUI Core 0 XAUI Core 1 Power PC Memory Controller Subsystem 4 TX PAIRS 4 RX PAIRS 4 TX PAIRS 4 RX PAIRS XTAL PLL XTAL PLL DUAL 10G PHY FMC PHY SFP+ PHY SFP+ TB FPGA 10G UDP Core 1 Power PC Memory Controller Subsystem 10G UDP Core 0 ‘XCTRL’ DDR2 SDRAM Image Buffers Storage DDR2 SDRAM QDRII SRAM Image Manipulation QDRII SRAM Aurora 64/66B XAUI QDRII Interface QDRII Interface LOCAL LINK Protocol QDRII SRAM Pixel reordering DDR2 Partial Images train buffer To Xpoint 10G Inputs from Detector

16. The TrainBuilder ATCA Data Acquisition Board for the European XFEL FE FPGA Firmware 16 Presented by John Coughlan. TWEPP Oxford. September 2012. Aurora Core 0 XPOINT 4 TX PAIRS 4 RX PAIRS 4 TX PAIRS 4 RX PAIRS Aurora Core 1 XAUI Core 0 XAUI Core 1 Power PC Memory Controller Subsystem 4 TX PAIRS 4 RX PAIRS 4 TX PAIRS 4 RX PAIRS XTAL PLL XTAL PLL DUAL 10G PHY FMC PHY SFP+ PHY SFP+ TB FPGA 10G UDP Core 1 Power PC Memory Controller Subsystem 10G UDP Core 0 ‘XCTRL’ DDR2 SDRAM Image Buffers Storage DDR2 SDRAM QDRII SRAM Image Manipulation QDRII SRAM Aurora 64/66B XAUI QDRII Interface QDRII Interface LOCAL LINK Protocol 10G Outputs to Farm QDRII SRAM Image Tiling DDR2 Full Images train buffer From Xpoint

17. The TrainBuilder ATCA Data Acquisition Board for the European XFEL TrainBuilder Firmware System 17 Presented by John Coughlan. TWEPP Oxford. September 2012. TrainBuilder FEMs

18. The TrainBuilder ATCA Data Acquisition Board for the European XFEL 18 Applications Presented by John Coughlan. TWEPP Oxford. September 2012.

19. The TrainBuilder ATCA Data Acquisition Board for the European XFEL Status First ATCA boards manufactured in Q1/2012 4 Boards all passed Boundary SCAN first time. Basic Tests working 10G links Tx and Rx DDR2 QDRII Crosspoint GbE controls Performance testing at RAL. Data Rates 1 GB/s/link? Soak Tests. Board in Hamburg for XFEL DAQ PC Farm tests since July. 19 Presented by John Coughlan. TWEPP Oxford. September 2012.

20. The TrainBuilder ATCA Data Acquisition Board for the European XFEL First Detector Application 20 1 x TB configured as 4 Inputs and 4 Outputs Presented by John Coughlan. TWEPP Oxford. September 2012. x4 Large Pixel Detector Prototype Quadrant. 4 FEMs 10G links

21. The TrainBuilder ATCA Data Acquisition Board for the European XFEL 2 x TBs : ½ Mpixel 21 Presented by John Coughlan. TWEPP Oxford. September 2012. 2 x TB System : 8 Links ½ Mpixel Passive Cables Tx & Rx x8 INPUT OUTPUT

22. The TrainBuilder ATCA Data Acquisition Board for the European XFEL FUTURE 4 x TBs : 1 Mpixel 2013 22 Presented by John Coughlan. TWEPP Oxford. September 2012. 1 Mpixel with 4 x ATCA TB Demonstrators. 16 LINKs x16 Parallel Opto Links SNAP 12 Tx / Rx Scaleable to larger systems NEW ATCA Switch Board NEW Optical RTMs @ 2.5 Gbps

23. The TrainBuilder ATCA Data Acquisition Board for the European XFEL Summary & Plans Developed an ATCA board for common XFEL DAQ Firmware infrastructure already operating incl. 10G UDP 2 x Demonstrator ATCA can instrument ½ Mpixel today Baseline is 1 Mpixel. Scheme exists to scale to larger detectors. Plans Complete Performance Tests on prototypes Develop Optical RTM and Switch card 2013 Integrate with 1 Mpixel detectors 2014 Install XFEL in 2015 Data taking at XEFL in 2016 Acknowledgements: M. Zimmer, I. Sheviakov (DESY Electronics) C. Youngman (XFEL DAQ) 23 Presented by John Coughlan. TWEPP Oxford. September 2012.

24. The TrainBuilder ATCA Data Acquisition Board for the European XFEL 24 Questions Presented by John Coughlan. TWEPP Oxford. September 2012.

25. The TrainBuilder ATCA Data Acquisition Board for the European XFEL 25 Spare Slides Presented by John Coughlan. TWEPP Oxford. September 2012.

26. The TrainBuilder ATCA Data Acquisition Board for the European XFEL European-XFEL Hamburg 26 Presented by John Coughlan. TWEPP Oxford. September 2012. 2D Camera captures 512 Images per Train Train repetition rate = 10 Hz ~5,000 fps Xray Free Electron Laser

27. The TrainBuilder ATCA Data Acquisition Board for the European XFEL Crosspoint Switch Operation 27 Presented by John Coughlan. TWEPP Oxford. September 2012. Switch Protocol : Xilinx Aurora 64/66B @ 2.5 Gbps Xpoint Switch changes only @ 10 Hz Partial Images 128 KB Time Ordered Full Image Trains 512 x 2 MB FEEs PC Farm DDR2

28. The TrainBuilder ATCA Data Acquisition Board for the European XFEL 10G UDP 28 LL FRAME (64 bit) UDP PKT STREAM FPGA FPGA - FPGA PC Tx : MAC, IP ADDR & PORT LUT (LL Frame Header) * Rx : IP, PORTS, MAC, Filter for PLAYBACK mode Software Programmable : Packet Length (Jumbo), IFG...etc *Plan to add Resends on Tx for “Reliable UDP” (TB to PC Farm) 10G PHY FPGA Presented by John Coughlan. TWEPP Oxford. September 2012. Local Link

29. The TrainBuilder ATCA Data Acquisition Board for the European XFEL PPC DDR2 Memory Controller 29 Presented by John Coughlan. TWEPP Oxford. September 2012. DDR2 SODIMM 64 bit 2 GB V5 FX100T 2 x PPC440 Hard IP blocks Each with 4 DMA engines Tx&Rx DDR2 controller 500 MT/s Local Link Interface 32 bit @ 250 MHz ~ 1 GB/sec Concurrent Read & Write LL Write LL Read Embedded C code manages DMA engines/buffers

30. The TrainBuilder ATCA Data Acquisition Board for the European XFEL TrainBuilder Next Generation 30 Presented by John Coughlan. TWEPP Oxford. September 2012. TB V7/K7 ≥8 x QSFP+ >8 x DDR3 SODIMM XP 144 x 144 @10G XP -> V7/K7 QDR Bandwidth? Bigger Packages No FMC