Frank Lemke Design concepts and measurements of the CBM DAQ network DPG – Frühjahrstagung Dresden 2013 HK 10.7 University of Heidelberg Computer Architecture Group Frank Lemke, Sven Schatral

Frank Lemke DPG – Frühjahrstagung Dresden Outline  The Compressed Baryonic Matter Experiment  Motivation  CBM Network Structure  Front-end ASIC Communication  FPGA and ASIC Based Readout Controllers  Conclusion

Frank Lemke 3 FAIR at GSI Darmstadt Germany  Facility for Antiproton and Ion Research (FAIR) extends the existing GSI accelerator and synchrotron  Starting 2018, FAIR will be used to measure atomic nuclei and the particles they are made up of ( January 2013 ) DPG – Frühjahrstagung Dresden 2013 ( )  Construction work has already begun and makes visible progress  Demonstrator prototypes have been build and development for the experiment readout systems are ongoing

Frank Lemke DPG – Frühjahrstagung Dresden The Compressed Baryonic Matter Experiment One of the eight FAIR experiments is the Compressed Baryonic Matter (CBM) experiment The CBM experiment  Investigates highly compressed nuclear matter using nucleus-nucleus collisions  Is a self-triggered detector system  Provides eight different kinds of detectors ( )

Frank Lemke DPG – Frühjahrstagung Dresden Motivation Create complete DAQ network solution usable within all stages of the network providing the required communication capabilities for CBM  Complete network approach allows design of straight forward and compact solutions fulfilling  Restricted space limits for hardware  High bandwidth requirement  Supporting CBM specific synchronization methodologies  No protocol conversion required  Easy for user adaption and providing reusable blocks for the CBM network

Frank Lemke 6 CBM Network Challenges DAQ network system needs to deliver  Flexibility for various build-up variants  Efficiency for data aggregation  Successive link rate increase  From 0.5 Gb/s up to 10Gb/s per lane  Clock distribution  Reliable system clock from one single master  Delivering up to 250 MHz over LVDS for stable FEE sampling  Precise time synchronization  In the order of the link bit clock cycles -> 2 ns  Support for centralized control system solutions  Dense interconnection solution  Handling of up to several TB/s of raw data DPG – Frühjahrstagung Dresden 2013

Frank Lemke CBM Network Structure 7DPG – Frühjahrstagung Dresden 2013 ( Walter F.J. Mueller, GSI, Darmstadt, 30 March 2012 )

Frank Lemke Front-end ASIC Challenges 8DPG – Frühjahrstagung Dresden 2013 Depending on type and system position FEE bandwidth varies  Starting at 0.5 Gb/s using one lane  Supporting up to four lanes with 2 Gb/s CBMnet has reusable design blocks  Control and status Register File (RF)  I2C support for RF access, debugging and bring-up  CTRL decode for CBMnet ctrl messages  ASIC SERDES implementations supporting synchronization  CBMnet protocol module with Master/Slave support  Unbalanced links: 1 up-stream and up to 4 down-stream links  Shift register chain and sub RF support for analog designed ASIC parts

Frank Lemke 9 Front-end ASIC Modules DPG – Frühjahrstagung Dresden 2013

Frank Lemke 10 CBMnet Implementation at SPADIC & STSXYTER DPG – Frühjahrstagung Dresden 2013 STSXYTER  Device will be used for the Silicon Tracking System  2 Gb/s read-out bandwidth  Currently in production SPADIC - Self-triggered Pulse Amplification and Digitization asIC ( Please visit  Device will be used for the transition radiation sub-detector (TRD)  Device to read-out and process small electrical detector signals on a single silicon  32-channel mixed signal  Self-triggered hit detection and neighbor readout  Full pulse recording  1 Gb/s read-out bandwidth  Current version 1.0 using CBMnet protocol communication and design modules

Frank Lemke SPADIC Measurement 11DPG – Frühjahrstagung Dresden 2013  Synchronization accuracy on bit clock level of 2ns was shown  Configuration registers have been written and control communication using CBMnet protocol was approved  Data was successfully streamed from the SPADIC

Frank Lemke FPGA Readout - Eval 12DPG – Frühjahrstagung Dresden 2013 It provides  SFP to use old readout chains  FMC adapter to attach specially developed board connecting multiple FEEs  HDMI cables used for first cabling  Identical FPGA type as new ROC used as emulation platform  Small and low cost beam time and laboratory readout chains  Limited synchronization capability Xilinx Spartan 6 eval-board SP605 with 45T device is used as first evaluation platform for the next generation FPGA Boards

Frank Lemke FPGA Readout – ROC3 13DPG – Frühjahrstagung Dresden 2013 In the collaboration a new universal readout controller is in production using a Xilinx Spartan6 150T device  FMC adapters for attaching FEE readout  Almost identical first readout using SP605 HDMI solution  Designed to provide the full synchronization feature set  Supporting up to 8 FEEs  Substitution of former concentrator and aggregator boards  Clock source or even multiplier  ECS/DCS emulator  Data sink and readout chain source  Emulation platform for HUB testing on limited small functional blocks HK 34.6: Modular CBM-ROC Firmware - Was bisher geschah und wie es weitergeht. SEBASTIAN MANZ und UDO KEBSCHULL ( Dienstag, 5. März, 15:30, HSZ-405)

Frank Lemke 14 ASIC Readout Challenges DPG – Frühjahrstagung Dresden 2013  Readout ASIC (HUB) needs to support at least 32 link FEE connections  Automatic and flexible initialization and synchronization of FEEs  Aggregation of data from numerous FEE devices  Rate conversion preserving reliable synchronization  Creating an excellent ratio for bandwidth per area  Up to 4x, 5 Gb/s for each lane, 20 Gb/s per HUB  3 HUBs bundled to 12x links with 60 Gb/s total  Specific SERDES designs  Standard cell SERDES to connect FEE devices (500Mb/s)  Full custom SERDES with CDR to connect back-end (at least 5Gb/s)

Frank Lemke 15 HUB ASIC Structure DPG – Frühjahrstagung Dresden 2013  Direct extraction and distribution of synchronization  Deadlock avoiding for data and control communication (VCs)  Large crossbar structures enabling flexible FEE attachment and fault tolerance  Fully remote control through RF  Flexible FEE re-initialization and control

Frank Lemke 16 Conclusion  The CBM protocol and the module concept have shown their usability  SPADIC front-end ASIC was successfully tested  Tests for STSXYTER are prepared and the chip was submitted  Basic FPGA prototyping for ROC usage and HUB ASIC emulations are ongoing The goal for 2013 is preparing the first HUB ASIC prototype design as miniASIC for testing of  Radiation tolerance of design parts  High-speed SERDES functionality  Data aggregation and synchronization functionalities DPG – Frühjahrstagung Dresden 2013

Frank Lemke Thank you for your attention ! Questions ? DPG – Frühjahrstagung Dresden 2013