Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Computer Instrumentation Introduction Jos Vermeulen, UvA / NIKHEF Topical lectures, 28 June 2007

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Computing essential ingredient of high-energy physics instrumentation for: trigger data-acquisition on-line monitoring experiment control calibration These lectures: Relevant hardware and software techniques, the ATLAS T/DAQ/DCS system serves as illustration

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June A Toroidal LHC ApparatuS (ATLAS): One of two general-purpose experiments at the Large Hadron Collider at CERN. Under construction in Point 1, opposite to the main entrance of CERN

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June SDX1 USA15 UX15 US15 Main control room Main Entrance CERN Point 1 A side (Geneva) C side (Jura) x y z

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Cut-away view of the ATLAS Detector

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 20076

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 20077

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June 20078

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June June 13, 2007: Lowering of side A End Cap Toroid (ECT)

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Descent through the access shaft

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Descent in the cavern

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Landed Vessel produced by Schelde Exotech, cold mass components by Brush HMA BV

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June The relation with computing …. June issue of ATLAS e-news ( Richard Stallman, the president and founder of the Free Software Foundation ….. went on a pit tour, where he had a chance to admire our freshly installed end-cap toroid!

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Computer instrumentation in high-energy physics: Detector Control System (DCS) Trigger Data-AcQuisition (DAQ)

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June ATLAS Detector Control System (DCS) LCS USAL1 LCS USAL2 LCS US15 LCS SDX1 LCS 1LCS 2LCS 3LCS 4 CoolingRacksEnvironELMB HEC HV Temp Barrel HV FE Crates HV LV Purity Front-End Systems Magnet CERN LHC DSS Data Viewer AlarmStatusWeb Operator Interface DCS_IS WAN CIC PixelSCTTRTLArMDTTGCRPCCSC LAN Tile Common Infrastructure Controls Local Control Stations Detector Safety System DCS Information Service Underground Subdetector Control Stations Embedded Local Monitor Box Global Control Stations

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June SDX1 USA15 UX15 ATLAS Trigger / DAQ DataFlow Overview

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Event data ≤ 100 kHz, 1600 fragments of ~ 1 kByte each ATLAS detector Read- Out Drivers ( RODs ) First- level trigger Read-Out Subsystems ( ROSs ) UX15 USA15 SDX1 USA15 UX15 Dedicated links Timing Trigger Control (TTC) 1600 Read- Out Links 10 Gigabit Ethernet ATLAS Trigger / DAQ DataFlow Overview RoI Builder Regions Of Interest VME ~160 PCs Data of events accepted by first-level trigger Event data requests Delete commands Requested event data ~40x10 ~320x1 Gbit/s ~20 switches

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Event data ≤ 100 kHz, 1600 fragments of ~ 1 kByte each ATLAS detector Read- Out Drivers ( RODs ) First- level trigger Read-Out Subsystems ( ROSs ) LVL2 Super- visor UX15 USA15 SDX1 CERN computer centre SDX1 USA15 UX15 Dedicated links Timing Trigger Control (TTC) 1600 Read- Out Links 10 Gigabit Ethernet ATLAS Trigger / DAQ DataFlow Overview RoI Builder DataFlow Manager Event Filter (EF) pROS ~ 870~1500 Regions Of Interest VME Data of events accepted by first-level trigger Event data requests Delete commands Requested event data stores LVL2 output dual 1, 2 or 4-core CPU nodes ~100~30 Network switches Event data pulled: partial ≤ 100 kHz, full ~ 3 kHz Event rate ~ 200 Hz Data storage Local Storage SubFarm Outputs (SFOs) LVL2 Farm + switches Network switches Event Builder SubFarm Inputs (SFIs) Second- level trigger ~40x10 ~320x1 Gbit/s ~20 switches ~160 PCs Gigabit Ethernet

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Event data ≤ 100 kHz, 1600 fragments of ~ 1 kByte each ATLAS detector Read- Out Drivers ( RODs ) First- level trigger Read-Out Subsystems ( ROSs ) UX15 USA15 Dedicated links Timing Trigger Control (TTC) 1600 Read- Out Links RoI Builder VME Data of events accepted by first-level trigger RODs, ROS PCs and ROBINs 10 Gigabit Ethernet Read-Out Drivers (ROD): subdetector-specific, collect and process data (no event selection) output via Read-Out Links (ROL, 200 MByte/s optical fibers) to buffers on ROBIN cards in Read-Out Subsystem (ROS) PCs Same type of ROLs, ROBINs and ROS PCs used for all sub-detectors ROBINs: 64-bit 66 MHz PCI-X cards 3 ROL inputs ROS PCs: 4U rack-mounted PCs with 4 ROBINs => 12 ROLs per ROS PC ~40x10 ~320x1 Gbit/s ~20 switches ~150 PCs

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Region of Interest (RoI) Builder receives for each first-level accept information from first-level trigger and passes formatted information to one of the LVL2 supervisors. LVL2 supervisor decides for one of the processors in the LVL2 farm and sends it the RoI information. LVL2 processor requests data from the ROSs as needed (possibly in several steps), produces an accept or reject and informs the LVL2 supervisor. Result of processing is stored in pseudo-ROS (pROS) for an accept. LVL2 supervisor passes decision to the DataFlow Manager. Trigger/DAQ DataFlow associated with second-level (LVL2) trigger First- level trigger Read-Out Subsystems ( ROSs ) LVL2 Super- visor USA15 Timing Trigger Control (TTC) 1600 Read- Out Links 10 Gigabit Ethernet RoI Builder Regions Of Interest Event data requests Delete commands Requested event data ~40x10 ~320x1 Gbit/s ~20 switches ~150 PCs SDX1 DataFlow Manager Event Filter (EF) pROS ~ 870~1500 dual 1, 2 or 4-core CPU nodes ~100~30 Network switches Event rate ~ 200 Hz Local Storage SubFarm Outputs (SFOs) LVL2 Farm + switches Network switches Event Builder SubFarm Inputs (SFIs)

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June For each accepted event the DataFlow Manager decides for a Sub-Farm Input (SFI) and sends it a request to take care of the building of a complete Event. The SFI sends requests to all ROSs for data of the event to be built. Completion of building is reported to the DataFlow Manager. For rejected events and for events for which event Building has completed the DataFlow Manager sends "clears" to the ROSs for up to 100 events together. On request the event data are passed from SFI to an Event Filter processor. Trigger/DAQ DataFlow associated with Event Building Event Building rate ~ kHz First- level trigger Read-Out Subsystems ( ROSs ) LVL2 Super- visor USA15 Timing Trigger Control (TTC) 1600 Read- Out Links 10 Gigabit Ethernet RoI Builder Regions Of Interest Event data requests Delete commands Requested event data ~40x10 ~320x1 Gbit/s ~20 switches ~150 PCs SDX1 DataFlow Manager Event Filter (EF) pROS ~ 870~1500 dual 1, 2 or 4-core CPU nodes ~100~30 Network switches Event rate ~ 200 Hz Local Storage SubFarm Outputs (SFOs) LVL2 Farm + switches Network switches Event Builder SubFarm Inputs (SFIs)

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Simple model ("paper model") used to predict average number of ROS PCs and ROLs from which data is needed for LVL2 trigger processing, for example for design luminosity trigger menu, per first-level accept: 16.2 ROLs or 8.4 ROS PCs -> RoI-driven processing is a key property of the ATLAS LVL2 system, but also makes the system more complex and its performance not so straight- forward to predict. With kByte per fragment need network bandwidth of ~ 2 GByte/s at 100 kHz first-level trigger accept rate (instead of ~ 150 GByte/s for full read-out at 100 kHz) for LVL2 traffic Second-Level (LVL2) Trigger and Event Building: rates Event Building rate ~ kHz (bandwidth ~ GByte/s) First- level trigger Read-Out Subsystems ( ROSs ) LVL2 Super- visor USA15 Timing Trigger Control (TTC) 1600 Read- Out Links 10 Gigabit Ethernet RoI Builder Regions Of Interest Event data requests Delete commands Requested event data Event rate ~ 200 Hz ~40x10 ~320x1 Gbit/s ~20 switches ~150 PCs SDX1 DataFlow Manager Event Filter (EF) pROS ~ 870~1500 dual 1, 2 or 4-core CPU nodes ~100~30 Network switches Event rate ~ 200 Hz Local Storage SubFarm Outputs (SFOs) LVL2 Farm + switches Network switches Event Builder SubFarm Inputs (SFIs)

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Application Specific Integrated Circuits (ASICs) Field Programmable Gate Arrays (FPGAs) Digital Signal Processors (DSPs) Microcontrollers Embedded microprocessors Crate processors Personal Computers (PCs) Compute servers Linux, multi-threaded C++, Java for GUIs usually no OS, cross development of software written in C or C++, exceptionally in assembler "programmed" in VHDL or Verilog + "fitting", specialists only, mostly electronic engineers, "firmware" can be changed in-circuit in many designs expensive, design by specialists, cannot be changed after production Computing hardware

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June VME bus: parallel bus in crates PCI, PCI-X bus : parallel bus in PCs PCI-Express: serial point-to-point connections inside PC CAN: serial bus system used for controls JTAG: serial connections between integrated circuits SPI, I2C: short distance serial connections RocketIO links of Xilinx FPGAs: up to Gbit/s serial point-to-point links for connecting FPGAs GOL: Gigabit Optical Link, developed by CERN (radiation hard sender) S-link: protocol for point-to-point links, developed by CERN TTC: Timing and Trigger Control system: for broadcasting trigger decisions, but can also transmit control information, developed by CERN Switched Ethernet: 100 Mbit, 1 Gigabit, 10 Gigabit Other switched network technology (not in ATLAS, but e.g. in CMS: Myrinet) Connection technology

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Interrupts Direct Memory Access (DMA) Memory-mapped I/O Error detection using parity bits or a Cyclic Redundancy Check (CRC) -> See presentation Sander Klous Memory management Drivers Multi-threaded programming Inter-process communication Remote process invocation Traffic shaping to avoid queueing Databases Techniques

Jos VermeulenTopical lectures, Computer Instrumentation, Introduction, June Tomorrow: examples and discussion of different types of computing hardware and interconnection technology and of relevant techniques in the context of: I.ATLAS DCS and front-end readout (TTC, on-detector, RODs) II.ATLAS Triggering and DAQ