1. Particle Physics & Astrophysics Representing: Mark Freytag Gunther Haller Ryan Herbst Michael Huffer Chris O’Grady Amedeo Perazzo Leonid Sapozhnikov Eric Siskind LCWS 2008 Data Acquisition and Global Detector Network SiD ATCA DAQ System Matt Weaver, weaver@slac.stanford.edu SLAC National Accelerator Laboratory November 18, 2008 IEEE 2008 Nuclear Science Symposium. Talk based largely on Huffer, et al., “ A new proposal for the construction of high speed, massively parallel, ATCA based Data Acquisition Systems,” IEEE 2008 Nuclear Science Symposium. Represents development work done at SLAC for the following projects: PetaCache, LCLS, LSST

2. Particle Physics & Astrophysics 2 Outline Modular DAQ design to address new projects ATCA demonstration boards – Reconfigurable Cluster Element – Cluster Interconnect Module Example application to SiD

3. Particle Physics & Astrophysics 3 Abstract HEP Data Acquisition System – Analog shaping – Digitization – Transmission – Trigger pipeline buffering – Charge injection Front-End Electronics Dataflow Detector Back-End Systems – High-Level Triggering – Event-Building – Archival – Data Quality Assurance – Feature Extraction – Compression – Re-formatting – Event buffering – Aggregation – Reception & Transmission – Protocol conversion emphasis is on this component

4. Particle Physics & Astrophysics 4 A cluster of 12 elements To back-end systems Cluster Interconnect Elements From Front-End systems switching fabric

5. Particle Physics & Astrophysics 5 Typical (5 slot) ATCA crate fans “back-end” I/O “front-end” I/O “interconnect” board Power supplies “element” board Shelf manager Front Rear

6. Particle Physics & Astrophysics 6 (Reconfigurable) Cluster Element (RCE) Bundled software: – bootstrap loader – Open Source kernel (RTEMS) POSIX compliant interfaces standard I/P network stack – exception handling support MGTs Core DSP tiles Combinatoric logic Resources Processor 450 MHZ PPC-405 512 MByte RLD-II Boot Options Memory Subsystem Configuration data 128 MByte Flash Data Exchange Interface (DEI) instruction reset & bootstrap options Class libraries ( C++) provide: – DEI support – configuration interface Bundled software: – GNU cross-development environment ( C & C++) – remote (network) GDB debugger – network console

7. Particle Physics & Astrophysics 7 Resources Multi-Gigabit Transceivers (MGTs) – up to 24 channels of: SER/DES input/output buffering clock recovery 8b/10b encoder/decoder 64b/66b encoder/decoder – each channel can operate up to 6.5 gb/s – channels may be bound together for greater aggregate speed Combinatoric logic gates flip-flops (block RAM) I/O pins DSP support – contains up 192 Multiple-Accumulate-Add (MAC) units

8. Particle Physics & Astrophysics 8 RCE board + RTM (Block diagram) MFD RCE flash memory RCE slice 0 slice 1 slice 2 slice 3 slice 7 slice 6 slice 5 slice 4 MFD Fiber-optic transceivers Zone2 PayloadRTM Zone3 E0 E1 E0 E1 Base network Fabric network

9. Particle Physics & Astrophysics 9 RCE board + RTM Media Carrier with flash Media Slice controller RCE Zone 1 (power) Zone 2 Zone 3 transceiver s RTM

10. Particle Physics & Astrophysics 10 Front-end Interface Example Transceiver Register Interface Clock/Reset generator Transmit Interface GT11CLK MGT Reference clock PGP protocol core MGT GT11CLK Trigger Interface reset request input[31:0] address[23:0] request R/W ACK status output[31:0] context opcode trigger enable SOF EOF error byte/word data[15:0] clock reset FEE logicFront-End Board FPGA POR

11. Particle Physics & Astrophysics 11 Cluster Interconnect board + RTM (Block diagram) MFD CI Q2 P2 1-GE 10-GE XFP Q0 Q1Q3 XFP PayloadRTM P3 10-GE 1-GE 10-GE XFP 10-GE (fabric) (base) base fabric (fabric) (base)

12. Particle Physics & Astrophysics 12 Cluster Interconnect board + RTM Zone 3 Zone 1 10 GE switch XFP 1G Ethernet Mgmt Processor XFP RTM

13. Particle Physics & Astrophysics 13 SiD Data Acquisition EMCal example – ~100M channels – 4bytes/hit x 4hits/train (KPiX) – readout in 100ms (1/2 of inter-train spacing) – EMCal total readout data rate ~ 128Gbps

14. Particle Physics & Astrophysics 14 SiD DAQ continued… to Cluster Element FPGA Protocol interface One ATCA crate EMCal (super ? -) concentrators FPGA 8 Gbytes/sec 500 Mbytes/sec 16 elements 8 RCE boards 2 CI boards 1.6 Gbytes/train Cluster Element Cluster Interconnect

15. Particle Physics & Astrophysics 15 Status SLAC is positioning itself for the production of a new generation of DAQ systems – strategy is based on the idea of modular building blocks inexpensive computational elements (the RCE) interconnect mechanism (the CI) industry standard packaging (ATCA) Architecture is now relatively mature – both demonstration boards (& their corresponding RTMs) are functional RTEMS ported & operating network stack fully tested and functional – performance and scaling have been demonstrated to meet expectations Current focus is on applications using these concepts… – LSST DAQ – LCLS DAQ – PetaCache project Application of same boards to SiD DAQ could yield a compact yet scalable system.

16. Particle Physics & Astrophysics 16