Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Computer Instrumentation Detector Control and Front-End Readout Jos Vermeulen, UvA / NIKHEF Topical lectures, 29 June 2007

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Detector Control

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June ATLAS Detector Control System (DCS) LCS USAL1 LCS USAL2 LCS US15 LCS SDX1 LCS 1LCS 2LCS 3LCS 4 CoolingRacksEnvironELMB HEC HV Temp Barrel HV FE Crates HV LV Purity Front-End Systems Magnet CERN LHC DSS Data Viewer AlarmStatusWeb Operator Interface DCS_IS WAN CIC PixelSCTTRTLArMDTTGCRPCCSC LAN Tile Common Infrastructure Controls Local Control Stations Detector Safety System DCS Information Service Underground Subdetector Control Stations Embedded Local Monitor Box Global Control Stations

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June GCS MDT TGCCSC….RPC EABA BCEC PS PS Global control stations (operator) All ATLAS subsystems ELMBJTAG… Sub-detector Control Station PSLCS PS PS Endcap A-side Endcap C-side Barrel A-side Barrel C-side Power supply Local Control Station ELMBJTAG…ELMBJTAG…ELMBJTAG… Readout setup B & T sensors V & I monitoring Alignment Example: DCS for the Monitored Drift Tube (MDT) chambers ResistivePlateChambers Thin Gap Chambers Cathode Strip Chambers CAN

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Block diagram ELMB128 CAN: Controller- Area-Network: implemented on twisted-pair wire ELMB: Embedded Local Monitor Box

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June CAN is a bus system: Controller ELMB Protocol supports arbitration for bus access, arbitration is based on comparisons made by each node of the signal sent to the bus and of the signal as available on the bus. Clocks of nodes synchronized by edges in signals, non-return to zero (NRZ) coding (during a bit cell the signal level represents either a 0 or a 1) with stuffing of a 0-bit after 5 1-s or a 1-bit after 5 0-s and removal of this bit by the receiver(s) guarantees a sufficient number of edges. -> typical speed 1 Mbit/s or lower

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June CAN CAN: international standard: ISO Application: automotive and industrial Software in ATLAS used in conjunction with CAN: CANOpen CANOpen: CAN-based higher layer protocol See e.g.: Protocol: see e.g.: OPC (Object linking and embedding for Process Control) client-server connection: industrial standard: see e.g.:

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June T-sensors 30x max  T  0.2 o C B-sensors 2x max  B/B  ELMB for the Monitored Drift Tube (MDT) chambers

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June JTAG (Joint Test Action Group) boundary scan: IEEE From “JTAG boundary scan started as a method of testing ICs and their interconnections using a shift register built into the chip so that inputs could be shifted in and the resulting outputs could be shifted out using only four I/O pins (clock, input data, output data, and state machine mode control). This eliminated the need for complex, expensive, bed-of-nails cards for low-speed probing of IC I/O pins. Eventually, the uses of JTAG expanded to include things like debugging software for embedded microcontrollers, thus reducing the need for in-circuit emulators. And JTAG is a natural match for downloading configuration bitstreams to FPGAs.” FPGA: Field Programmable Gate Array: logic elements in a single chip that are configured by e.g. bits in a static RAM, bit patterns typically are loaded via JTAG

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June SPI: Serial Peripheral Interface: synchronous serial connection, low-cost, speed up to ~ Mbit/s MasterSlave Clock Data Select Data sampled on edges of clock signal

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June ELMB128 The board is placed on a motherboard Processor: Atmel ATmega 128L 8-bits microcontroller with 128 kByte FLASH memory, 4 kByte static RAM, 4 kByte of EEPROM, runs at 4 MHz clock speed Programmed in C CAN bus controller optocouplers transceiver and protocol chip 16-bit ADC 64-channelmultiplexer

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Block diagram ATmega 128 No external data and address bus, but port A and C can be configured as data/address bus I/O via parallel ports Vectored interrupts

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Microcontroller has small bootloader program in flash memory, which uses port B, pin1 - 3, for loading a program via the CAN bus. CAN bus controller has an SPI (Serial Peripheral Interface) connection

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Interrupt: indication that a certain condition has arisen, in the form of a signal to a CPU. The signal “interrupts” the processor, which in the simplest case saves its program counter on the stack and starts execution of an interrupt service routine. The routine can be at a fixed location in memory, or at a location stored in an entry of a “vector table”, which itself is stored in a well-defined place. The latter possibility (“vectored interrupts”) allows to execute different service routines for different signals. After a “return from interrupt” instruction processing continues at the place where program execution was interrupted. The signal can be an external signal, in the case of an "edge-sensitive" interrupt a change in the level of an input can give rise to an interrupt, for a "level-sensitive" interrupt the level of the signal determines whether an interrupt will be caused. An interrupt may also be due to an internal signal, e.g. caused by division by zero or by executing a special instruction. In the first case one usually refers to an "exception", in the latter case, which e.g. provides a mechanism for a running process to invoke a process scheduler, one usually refers to a “trap”.

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Memory-mapped I/O: input by reading from a certain address, or output by writing to an address. E.g. in the ATmega 128 each parallel port has 3 words at fixed locations associated with it, the bits of one determine for each of the 8 pins whether the pin behaves as input or output, the bits of the second word can be used for setting the state of the pin / pins configured as output, while the last word can be used for reading the state of the pin /pins configured as input. (NB in the ATmega 128 the addresses depend on the type of read or write instruction used, see: Polled I/O versus interrupt-driven I/O: by inspecting the states of bits using memory-mapped I/O it is possible to detect changes in these bits and react to these changes. Regular inspection of bits is also know as "polling". Alternatively, interrupts provide automatic change detection, but a running program can be temporarily halted at any moment, which may require special precautions and for some applications cannot be allowed. Polling is strictly deterministic with respect to time of occurrence and there is almost no overhead in detecting a change, while the jump into an interrupt service routine requires saving of context (in any case of register contents) and restoring the context after servicing the interrupt, i.e. more CPU cycles are needed. One has to take this into account when choosing between polled I/O and interrupt-driven I/O.

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Hardware memory management: so far it has been assumed that the software has unlimited access to all addresses and can react directly to interrupt signals. This may be true for microcontrollers like the ATmega 128 or for DSPs, but in general this is not the case for more complex processors equipped with memory management hardware and running an operating system (e.g. Linux). The memory management hardware is controlled by the operating system and translates the addresses generated by user software into physical addresses. However, the physical addresses must lie within certain windows, set up by the operating system and depending on the process generating these addresses. Any address outside the window gives rise to an exception, which usually leads to a crash of the process. This prevents direct memory-mapped I/O, in general "drivers" have to take care of the I/O, and user software has to control these drivers. However, it is also possible to open a window on the address region to be used for I/O by a driver and make direct access possible for user processes. In this way user processes running on the VME crate processor modules in the ROD crates are given access to the modules in the crate. Interrupts are usually serviced by the operating system, but can for example be transformed into signals for C/C++ programs

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June ADC 24-bit ID B-sensor 2 ADC 24-bit ID MDT Chamber Magnetic Field Sensors (B x, B y, B z and T ) CAN-bus ELMB micro CAN B-sensor 1 MDT-DCS module 24-bit ADC 24-bit 16-bit ADC SPI 7 4 B-sensor 0 ID MDT Front-end Electronics (CSM) JTAG: electronics configuration DIG-I/O 3 Analog inputs (Voltages & Temperatures 64 channels) 16-bit ADC DIG-I/O 4 4 control/status functions JTAG CSM-ADC 5 (ca. 600 chambers with one, two or four B-sensor modules each) NTC Temperature Sensors (10 to 20 per chamber, 30 max) SPI-AUX 5 (ca chambers in total) MDT/ ATLAS DCS (CANopen) + power (to next node) B-sensor 3 (spare) SPI ca. 3m MDTs: on-chamber support for controls and monitoring

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June MDT chambers (400,000 tubes, barrel+endcaps) correspond to: ­ 1200 Front end electronics boards ­ 1200 ELMBs ­ 2400 HV channels ­ 600 LV channels ­ 10,000 Temperature sensors ­ 1000 Magnetic field sensors ­ 10,000 Optical hardware components ­ 15,000 different Optical Alignment images Need to deal with this in an organized way: use a SCADA system

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June From “SCADA stands for Supervisory Control And Data Acquisition and these are commercial software systems used extensively in industry for the supervision and control of industrial processes. The standard SCADA functionality can be summarized as follows: ･ Data acquisition ･ Data logging and archiving ･ Alarm handling ･ Access control mechanism ･ Human Machine Interface including many standard features e.g. alarm display, trending PVSS is an industrial SCADA product from the Austrian company ETM. PVSS has the following strengths that make it interesting in the HEP domain: ･ It can run in a distributed manner with any of its manager running in a distributed manner ･ It is possible to integrate distributed systems ･ It has multi-platform support (Linux and Windows) ･ It is device oriented with a flexible data point concept ･ It has advanced scripting capabilities ･ It has a flexible API allowing access to all features of PVSS from an external application”

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June MDT DCS Alignment On-chamber Off-chamber RasCaM * (5452x) RasLED * (6672x) mask * (6672x) lens * (6672x) RasMuX (768x) Prototype: MiniMasterMuX I 2 C, video, clock, power power JTAG, video, clock Master MuX USA15 MuX (48x) (1 crate) 16 RASNIK (Red Alignment System NIKHEF) NB: slide may not be completely up-to-date

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June I2C: Inter-IC bus: low-speed (O(1 Mbit/s) ), low-cost Two wires: serial data (SDA) and serial clock (SCL) Devices connected are addressable and all can become master -> like for CAN, protocol supports arbitration

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June compute servers continuously analyze images

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Front-End Readout

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June ATLAS detector Read- Out Drivers ( RODs ) First- level trigger Dedicated links 1600 Read- Out Links TTC First-level trigger accepts cause transfer of data from on-detector buffers to RODs TTC broadcasts LVL1 accepts via tree of optical fibers, can also transmit other messages

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June TTC protocol LHC bunch crossing time “Biphase mark coding”: a “1” is marked by a transition in the middle of a cell, the clock needed for sampling can be retrieved from the signal received

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Example: the MDT front-end readout system

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Gbit/s optical fibre, S-Link to ROB CSM link (GOL) MDT Read- Out Driver (MROD) Chamber 0.8 or 1.6 Gbit/s optical fibre 80 Mbit/s CSM link (GOL) In USA15 24 ch. TDC Chamber Service Module (CSM) up to ch. TDC ROL 24 ch. TDC Chamber Service Module (CSM) up to ch. TDC 0.8 or 1.6 Gbit/s optical fibre ASICs and FPGAsFPGAs and DSPs MDT front-end readout ~1200 chambers 204 MRODs

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June On the chambers there is only processing of the data in the TDCs In the TDCs (ASICs) time stamps within a time window corresponding to the maximum drift time in the chambers are extracted and formatted. The TDC data is multiplexed by an FPGA onto a GOL (Gigabit Optical Link). The GOL sender chip is a radiation hard device, developed by CERN. The link uses 8B/10B encoding, which is commonly used for fast serial links: 10 bits are used for encoding 8 bits, so that the resulting signal cannot have a DC component and has enough transitions to recover a clock signal. 8B/10B encoding allows the transmission of "control tokens", i.e. of 10-bit patterns that do not correspond to possible 8-bit patterns. Successive chamber readout cycles (in one readout cycle per TDC one word - if available, otherwise a placeholder word is substituted - is transmitted) are separated by such control tokens on the link between CSM and MROD. Original paper on 8B/10B encoding, IBM Journal of Research, 1983:

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June MRODin SHARC E Memory CSM Memory MRODin Memory CSM Memory MRODin SHARC C Memory CSM Memory SHARC A Event Builder & SLINK interface TIM (TIM module receives TTC information MRODout VME64x ReadOut Link (ROL) VME interface TIM interface RocketIO link FPGA SHARC D SHARC link Bus SHARC link FPGA Bus MROD block diagram

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Field Programmable Gate Arrays: two important categories: Static RAM (SRAM) based: reconfigurable: bits in SRAM determine interconnection patterns and in look-up tables (LUTs) logic functions. Complex circuits possible, processors can be embedded in the FPGA, as well as other dedicated parts, e.g. interfaces for high speed links. Usually the "configuration" of the FPGA is stored in on-board flash memory and is loaded into the FPGA at power-up. The functionality of the circuit can be modified by loading a modified configuration into the flash memory. SRAM bits may flip, in particular due to radiation: reloading the configuration is then necessary, unless redundant circuitry and/or special techniques are used Altera, Xilinx Antifuse based: by blowing "antifuses" low resistance links are produced (hence the name antifuse) Only once programmable, but no bit flips in SRAM determining configuration: radiation hard Actel

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June MRODin: XC2VP7 MRODout: XC2VP20 PowerPC processor blocks not used Xlinix FPGAs used in MROD

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Example of complex circuitry that can be implemented in an FPGA: MRODin FPGA

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June The Xilinx FPGAs used in the MROD are equipped with RocketIO links: bi-directional serial link, transfer speed up to Gbit/s, 8 per FPGA 8B/10B coding Used for servicing CSM - MROD links and for communication between FPGAs FPGA technology allows to implement complex and performant dedicated functionality in a single device, modern devices support fast inter-device connects

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June SHARC DSP ADSP Digital Signal Processor (SHARC) 512 kByte on-chipSRAM

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Handshaking between transmitter and receiver SHARC links

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June Booting SHARCs: by writing code directly to internal memory (MRODout SHARC, via VME bus) or via link (MRODin SHARCs) using the link interfaces of the MRODout SHARC, which are directly accessible from the VME bus SHARCs communicate via links, MRODout SHARC via VME-bus with crate processor (PowerPC processor running Linux) SHARC software in C++: initialisation of MROD, error detection (using interrupts generated by FPGAs), monitoring (correctness of event data) Direct Memory Access (DMA): data transfer under control of a DMA controller, the CPU only takes care of initializing the controller by supplying it a source address, a destination address and a length of a block to be transferred and then starts the transfer by setting a bit in the controller. Once finished the controller can generate an interrupt if requested to do so, or the end of the transfer can be detected with the help of polling on a bit. The DMA controllers of the SHARC also support "chained transfers": a number of transfers can be done without intervention of the CPU by setting up the necessary information in memory in the form of a linked list and pointing the DMA controller to thead of the list.

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June S-link is a specification of data link interfaces, where the link transports data mainly in one direction and supports flow control (XON/XOFF mechanism) ATLAS: physical link: bi-directional optical (fiber pair), data throughput (net): 200 MByte/s, 8B/10B coding, CRC error check Successive even fragments are separated by control tokens on the link. Figure from H.C. van der Bij et al., S-LINK: A Prototype of the ATLAS Read-out Link, Presented at the Fourth Workshop on Electronics for LHC Experiments, Rome, September 1998) S-link

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June VME bus: parallel (up to 64 bit) multi-master asynchronous bus for use in crate. Most ROD crates in ATLAS use VME64x. VME dates back to The bus is not very fast with respect to present day standards (up to 160 MByte/s), the standardized mechanics are convenient, as well as the vertical cooling and relatively high power dissipation possible For more information: (access to standards documents only from inside CERN domain)

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June MROD

Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June ROL 9U VME crate Crate processor