1. Data Acquisition Systems for Future Calorimetry at the International Linear Collider Matt Warren, on behalf of CALICE-UK Collaboration

2. 2 29 Oct 07Matt Warren - DAQ for Calorimetry at ILC Introduction Building a DAQ for multiple ILC CAL sub-detector prototypesBuilding a DAQ for multiple ILC CAL sub-detector prototypes –Paying attention real ILC environment –EUDET Testbeam in 2009! (EU funded DETector project shares much of CALICE) For economies of scale, we are attempting a generic DAQ for many (even SLHC??)For economies of scale, we are attempting a generic DAQ for many (even SLHC??) –Modular/Generic Structure: Generic readout system as much as possibleGeneric readout system as much as possible Detector specific interfaces only at ends of chainDetector specific interfaces only at ends of chain Other ‘bespoke’ functionality in firmwareOther ‘bespoke’ functionality in firmware –Commercial components and protocols where possible Readout links use standard connectors and protocolsReadout links use standard connectors and protocols Based on PCs with PCIe cards (“backplaneless”)Based on PCs with PCIe cards (“backplaneless”) Clock and Control attempts commercial hardware tooClock and Control attempts commercial hardware too –Extract clock and ‘fast’ signals from commercial signalling Software generic for all detectorsSoftware generic for all detectors –Try use something off-the-shelf … BUT first, an introduction to the ILC CAL …

3. 3 29 Oct 07Matt Warren - DAQ for Calorimetry at ILC ILC Calorimetry ILC Calorimetry is dense and high granularityILC Calorimetry is dense and high granularity –squeezed between large tracker & expensive coil –>100M channels –No room for electronics or cooling. Bunch structure interesting:Bunch structure interesting: –~200ms gaps between bunch-trains –Trains 1ms long, 300ns bunch spacing Triggerless – sample data from every bunch- crossingTriggerless – sample data from every bunch- crossing SO (the problem): 100M channels, analog signals100M channels, analog signals = front-end electronics inside detector Results in high power densityResults in high power density –but no room for cooling Long gap allows chips a 1% duty cycleLong gap allows chips a 1% duty cycle Solution: Power Pulsing Time structure of bunches Trains of bunches Individual bunches ECAL HCAL M. Anduze

4. 4 29 Oct 07Matt Warren - DAQ for Calorimetry at ILC Sub-Detector Geometries (+ASICs) ASICS Must share readout resource (daisy chain) Bunch rate too high for instantaneous data transfer. Too much chip resource to store all events SO: ‘Auto-trigger’ – store only data over-threshold with pad id + (bunch-number)‘Auto-trigger’ – store only data over-threshold with pad id + (bunch-number) <5kByte / bunch-train/ASIC<5kByte / bunch-train/ASIC HCAL half-octant ECAL Module-0 (reduced-Z octant) L = 150 cm ASIC (>100 in total!) Detector Unit (e.g. ECAL Slab) M. Anduze

5. 5 29 Oct 07Matt Warren - DAQ for Calorimetry at ILC DAQ architecture Detector Unit: Sensors & ASICs DIF: Detector InterFace -connects generic DAQ and services LDA: Link/Data Aggregator – fanout/in DIFs & drive link to ODR ODR: Off Detector Receiver – PC interface for system. C&C: Clock & Control: Fanout to ODRs (or LDAs) LDA Host PC PCIe ODR Host PC PCIe ODR Detector Unit DIF C&C Detector Unit DIF Detector Unit DIF Detector Unit DIF Storage 1-3Gb Fibre 50-150 Mbps HDMI cabling 10-100m 0.1-1m Detector Counting Room

6. 6 29 Oct 07Matt Warren - DAQ for Calorimetry at ILC DIF (Detector InterFace) FPGA + detector hardware connected to Detector Unit.FPGA + detector hardware connected to Detector Unit. Two halves – Generic DAQ and Specific DetectorTwo halves – Generic DAQ and Specific Detector –3 detectors: ECAL, AHCAL, DHCAL –1 DAQ Interface! Focusing on the DAQ side: From LDA, receive, decode/regenerate and distribute clocks, fast commands, config data and slow controls.From LDA, receive, decode/regenerate and distribute clocks, fast commands, config data and slow controls. From ASICs, receive, buffer, package and forward data to LDAFrom ASICs, receive, buffer, package and forward data to LDA –ASICs power-up and read-out power-down in turn ALSO: USB interfaceALSO: USB interface –Hardware designers already have one –DAQ plans to integrate for stand-alone testing Detector Unit DIF USB  DAQ e.g. ECAL M. Goodrick

7. 7 29 Oct 07Matt Warren - DAQ for Calorimetry at ILC DIF-LDA link Serial links running at multiple of machine clockSerial links running at multiple of machine clock ~50Mbps (raw) bandwidth minimum~50Mbps (raw) bandwidth minimum robust encoding (8B/10B)robust encoding (8B/10B) HDMI cables/connectors interface.HDMI cables/connectors interface. –Commercially available cables –Rated >300Mb –Even halogen free available Signals (ideally just TX/RX but …):Signals (ideally just TX/RX but …):  Clock (diff)  Control/Fast (diff)  Very Fast (diff)  Data (diff)  single ended aux x2 (or UTP) LDAs serve even/odd DIFs for redundancyLDAs serve even/odd DIFs for redundancy LDA Detector Unit DIF Detector Unit DIF Detector Unit DIF Detector Unit DIF

8. 8 29 Oct 07Matt Warren - DAQ for Calorimetry at ILC LDA (Link/Data Aggregator) Located as close as possible to DIFsLocated as close as possible to DIFs –Shortest cables, but convenient location (space, cooling) Supports as many DIFs as possible considering bandwidth and physical constrainsSupports as many DIFs as possible considering bandwidth and physical constrains –Ideally 50 (20Mbps/DIF) –Prototype will have 10 Aggregates front-end data and sends it off-detectorAggregates front-end data and sends it off-detector –Fibre optic link. 1-3Gbps, with SFP (see next slide) Fanout C+C+C to DIFsFanout C+C+C to DIFs USB interface for stand-alone/top-of-chain testingUSB interface for stand-alone/top-of-chain testing LDA Currently using a commercial FPGA dev-board:Currently using a commercial FPGA dev-board: Enterpoint Broaddown2 – Xilinx Spartan3-200 With add-on boards for our needsWith add-on boards for our needs –SPF+SerDes for ODR link –10 HDMI connectors with clock fanout

9. 9 29 Oct 07Matt Warren - DAQ for Calorimetry at ILC ODR (Off Detector Receiver) + Link Host PC PCIe ODR Storage C&C Hardware: Using commercial FPGA dev-board:Using commercial FPGA dev-board: –PLDA XPressFX100 –Xilinx Virtex 4, 8xPCIe, 2x SFP (3 more with expansion board) Our own firmware and Linux driver softwareOur own firmware and Linux driver software SFPs for optic link Expansion (e.g. 3xSFP) Receives module data from LDAReceives module data from LDA –PCI-Express card, hosted in PC. –1-4 links/card (or more), 1-2 cards/PC –Buffers and transfers to store as fast as possible Fibre optic link to detector via SFP modules (std networking hw)Fibre optic link to detector via SFP modules (std networking hw) –Currently GigE (1.25Gb), but could higher and use different proto. Sends controls and config to LDA for distribution to DIFsSends controls and config to LDA for distribution to DIFs Interfaces to C+C for synchro runningInterfaces to C+C for synchro running –Goal to send clock and prompt controls over optic link too –Reset and reprog FPGAs Performance studies & optimisation on-going (see next slide):Performance studies & optimisation on-going (see next slide): –Bottleneck in writing data to disk.

10. 10 29 Oct 07Matt Warren - DAQ for Calorimetry at ILC ODR Throughput Measurements Ethernet frame size NDG – Network Data Generator IDG – Internal (ODR) Data Generator All measurements: single requester thread, single IO thread (disk write),All measurements: single requester thread, single IO thread (disk write), Each event fragment written to a separate file. Data written to the localdisk (fs: ext3)Each event fragment written to a separate file. Data written to the localdisk (fs: ext3) e.g. WORST CASE! ` NDG plot – between two separate machines, Gigabit, copper Eth on both sides Problem with test! A. Misiejuk

11. 11 29 Oct 07Matt Warren - DAQ for Calorimetry at ILC Clock & Control C&C unit provides machine clock and fast signals to 8x ODR/LDA.C&C unit provides machine clock and fast signals to 8x ODR/LDA. Logic control (FPGA, connected via USB)Logic control (FPGA, connected via USB) –Command encoders –Remote signal enable, clock selection –But capable of stand-alone, dumb mode Provision for async scintillator type signals (VFast)Provision for async scintillator type signals (VFast) LDA provides next stage fanout to DIFsLDA provides next stage fanout to DIFs –Eg C&C unit -> 8 LDAs -> 10 DIFs = 80 DUs. Signalling over same HDMI type cablingSignalling over same HDMI type cabling Facility to generate optical link clock (~125- 250MHz from ~50MHz machine clock)Facility to generate optical link clock (~125- 250MHz from ~50MHz machine clock) Commercial systems are not ideal here.Commercial systems are not ideal here. –Looking at custom protocol on fibre optic link –Prompt signals and low jitter clock recovery needs further investigation LDA Host PC PCIe ODR C&C Host PC PCIe ODR Machine Run- Control

12. 12 29 Oct 07Matt Warren - DAQ for Calorimetry at ILC Software and Operation Software: Looking for OTS software to cover both slow and fast controls: Early days – examining EPICs, ACE and DOOCS (the current favourite) - DOOCS is open source, actively developed, slow and fast controls, and already used by ILC community Operation: Two modes: 1) Configure: PCs controlled over network to send configuration to LDAs and DIFs 2) Run: –LDA/DIF set to data-taking mode –ODR configured for data reception, control handed to central. OR ODR needs no control – simply waits –Bunch-train starts/stop signals sent to LDAs control data flow. Fairly autonomous system (i.e. no trigger!)

13. 13 29 Oct 07Matt Warren - DAQ for Calorimetry at ILC Summary (an example – AHCAL) LDA DIF Detector Unit Off- Detector DAQ P. Göttlicher, DESY