1. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 20071 Computer Instrumentation Detector Control and Front-End Readout Jos Vermeulen, UvA / NIKHEF Topical lectures, 29 June 2007

2. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 20072 Detector Control

3. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 20073 ATLAS Detector Control System (DCS) LCS USAL1 LCS USAL2 LCS US15 LCS SDX1 LCS 1LCS 2LCS 3LCS 4 CoolingRacksEnvironELMB HEC HV Temp Barrel HV FE Crates HV LV Purity Front-End Systems Magnet CERN LHC DSS Data Viewer AlarmStatusWeb Operator Interface DCS_IS WAN CIC PixelSCTTRTLArMDTTGCRPCCSC LAN Tile Common Infrastructure Controls Local Control Stations Detector Safety System DCS Information Service Underground Subdetector Control Stations Embedded Local Monitor Box Global Control Stations

4. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 20074 GCS MDT TGCCSC….RPC EABA BCEC PS PS Global control stations (operator) All ATLAS subsystems ELMBJTAG… Sub-detector Control Station PSLCS PS PS Endcap A-side Endcap C-side Barrel A-side Barrel C-side Power supply Local Control Station ELMBJTAG…ELMBJTAG…ELMBJTAG… Readout setup B & T sensors V & I monitoring Alignment Example: DCS for the Monitored Drift Tube (MDT) chambers ResistivePlateChambers Thin Gap Chambers Cathode Strip Chambers CAN

5. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 20075 Block diagram ELMB128 CAN: Controller- Area-Network: implemented on twisted-pair wire ELMB: Embedded Local Monitor Box

6. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 20076 CAN is a bus system: Controller ELMB Protocol supports arbitration for bus access, arbitration is based on comparisons made by each node of the signal sent to the bus and of the signal as available on the bus. Clocks of nodes synchronized by edges in signals, non-return to zero (NRZ) coding (during a bit cell the signal level represents either a 0 or a 1) with stuffing of a 0-bit after 5 1-s or a 1-bit after 5 0-s and removal of this bit by the receiver(s) guarantees a sufficient number of edges. -> typical speed 1 Mbit/s or lower

7. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 20077 http://www.kvaser.com/can/protocol/main.htm CAN CAN: international standard: ISO 11898 Application: automotive and industrial Software in ATLAS used in conjunction with CAN: CANOpen CANOpen: CAN-based higher layer protocol See e.g.: http://www.can-cia.org/canopen/ Protocol: see e.g.: http://www.can-cia.org/can/ OPC (Object linking and embedding for Process Control) client-server connection: industrial standard: see e.g.: http://www.opcfoundation.org/Default.aspx/01_about/01_whatis.asp

8. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 20078 T-sensors 30x max  T  0.2 o C B-sensors 2x max  B/B  10 -4 http://www.nikhef.nl/pub/departments/ct/po/html/MDT/MDT-DCS-CANnode.pdf ELMB for the Monitored Drift Tube (MDT) chambers

9. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 20079 JTAG (Joint Test Action Group) boundary scan: IEEE 1149.1 From http://www.xess.com/faq/M0000297.HTM “JTAG boundary scan started as a method of testing ICs and their interconnections using a shift register built into the chip so that inputs could be shifted in and the resulting outputs could be shifted out using only four I/O pins (clock, input data, output data, and state machine mode control). This eliminated the need for complex, expensive, bed-of-nails cards for low-speed probing of IC I/O pins. Eventually, the uses of JTAG expanded to include things like debugging software for embedded microcontrollers, thus reducing the need for in-circuit emulators. And JTAG is a natural match for downloading configuration bitstreams to FPGAs.” FPGA: Field Programmable Gate Array: logic elements in a single chip that are configured by e.g. bits in a static RAM, bit patterns typically are loaded via JTAG

10. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200710 SPI: Serial Peripheral Interface: synchronous serial connection, low-cost, speed up to ~ 10 - 50 Mbit/s MasterSlave Clock Data Select Data sampled on edges of clock signal http://www.embedded.com/shared/printableArticle.jhtml?articleID=9900483

11. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200711 ELMB128 The board is placed on a motherboard Processor: Atmel ATmega 128L 8-bits microcontroller with 128 kByte FLASH memory, 4 kByte static RAM, 4 kByte of EEPROM, runs at 4 MHz clock speed Programmed in C CAN bus controller optocouplers transceiver and protocol chip 16-bit ADC 64-channelmultiplexer

12. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200712 Block diagram ATmega 128 No external data and address bus, but port A and C can be configured as data/address bus I/O via parallel ports Vectored interrupts

13. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200713 Microcontroller has small bootloader program in flash memory, which uses port B, pin1 - 3, for loading a program via the CAN bus. CAN bus controller has an SPI (Serial Peripheral Interface) connection

14. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200714 Interrupt: indication that a certain condition has arisen, in the form of a signal to a CPU. The signal “interrupts” the processor, which in the simplest case saves its program counter on the stack and starts execution of an interrupt service routine. The routine can be at a fixed location in memory, or at a location stored in an entry of a “vector table”, which itself is stored in a well-defined place. The latter possibility (“vectored interrupts”) allows to execute different service routines for different signals. After a “return from interrupt” instruction processing continues at the place where program execution was interrupted. The signal can be an external signal, in the case of an "edge-sensitive" interrupt a change in the level of an input can give rise to an interrupt, for a "level-sensitive" interrupt the level of the signal determines whether an interrupt will be caused. An interrupt may also be due to an internal signal, e.g. caused by division by zero or by executing a special instruction. In the first case one usually refers to an "exception", in the latter case, which e.g. provides a mechanism for a running process to invoke a process scheduler, one usually refers to a “trap”.

15. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200715 Memory-mapped I/O: input by reading from a certain address, or output by writing to an address. E.g. in the ATmega 128 each parallel port has 3 words at fixed locations associated with it, the bits of one determine for each of the 8 pins whether the pin behaves as input or output, the bits of the second word can be used for setting the state of the pin / pins configured as output, while the last word can be used for reading the state of the pin /pins configured as input. (NB in the ATmega 128 the addresses depend on the type of read or write instruction used, see:www.atmel.com/atmel/acrobat/doc2467.pdf) Polled I/O versus interrupt-driven I/O: by inspecting the states of bits using memory-mapped I/O it is possible to detect changes in these bits and react to these changes. Regular inspection of bits is also know as "polling". Alternatively, interrupts provide automatic change detection, but a running program can be temporarily halted at any moment, which may require special precautions and for some applications cannot be allowed. Polling is strictly deterministic with respect to time of occurrence and there is almost no overhead in detecting a change, while the jump into an interrupt service routine requires saving of context (in any case of register contents) and restoring the context after servicing the interrupt, i.e. more CPU cycles are needed. One has to take this into account when choosing between polled I/O and interrupt-driven I/O.

16. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200716 Hardware memory management: so far it has been assumed that the software has unlimited access to all addresses and can react directly to interrupt signals. This may be true for microcontrollers like the ATmega 128 or for DSPs, but in general this is not the case for more complex processors equipped with memory management hardware and running an operating system (e.g. Linux). The memory management hardware is controlled by the operating system and translates the addresses generated by user software into physical addresses. However, the physical addresses must lie within certain windows, set up by the operating system and depending on the process generating these addresses. Any address outside the window gives rise to an exception, which usually leads to a crash of the process. This prevents direct memory-mapped I/O, in general "drivers" have to take care of the I/O, and user software has to control these drivers. However, it is also possible to open a window on the address region to be used for I/O by a driver and make direct access possible for user processes. In this way user processes running on the VME crate processor modules in the ROD crates are given access to the modules in the crate. Interrupts are usually serviced by the operating system, but can for example be transformed into signals for C/C++ programs

17. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200717 ADC 24-bit ID B-sensor 2 ADC 24-bit ID MDT Chamber Magnetic Field Sensors (B x, B y, B z and T ) CAN-bus ELMB micro CAN B-sensor 1 MDT-DCS module 24-bit ADC 24-bit 16-bit ADC SPI 7 4 B-sensor 0 ID MDT Front-end Electronics (CSM) JTAG: electronics configuration DIG-I/O 3 Analog inputs (Voltages & Temperatures 64 channels) 16-bit ADC DIG-I/O 4 4 control/status functions JTAG CSM-ADC 5 (ca. 600 chambers with one, two or four B-sensor modules each) NTC Temperature Sensors (10 to 20 per chamber, 30 max) SPI-AUX 5 (ca. 1200 chambers in total) MDT/ ATLAS DCS (CANopen) + power (to next node) B-sensor 3 (spare) SPI ca. 3m MDTs: on-chamber support for controls and monitoring

18. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200718 1200 MDT chambers (400,000 tubes, barrel+endcaps) correspond to: ­ 1200 Front end electronics boards ­ 1200 ELMBs ­ 2400 HV channels ­ 600 LV channels ­ 10,000 Temperature sensors ­ 1000 Magnetic field sensors ­ 10,000 Optical hardware components ­ 15,000 different Optical Alignment images Need to deal with this in an organized way: use a SCADA system

19. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200719 From http://itcobe.web.cern.ch/itcobe/Services/Pvss/whatAreScadaAndPvss.html “SCADA stands for Supervisory Control And Data Acquisition and these are commercial software systems used extensively in industry for the supervision and control of industrial processes. The standard SCADA functionality can be summarized as follows: ･ Data acquisition ･ Data logging and archiving ･ Alarm handling ･ Access control mechanism ･ Human Machine Interface including many standard features e.g. alarm display, trending PVSS is an industrial SCADA product from the Austrian company ETM. PVSS has the following strengths that make it interesting in the HEP domain: ･ It can run in a distributed manner with any of its manager running in a distributed manner ･ It is possible to integrate distributed systems ･ It has multi-platform support (Linux and Windows) ･ It is device oriented with a flexible data point concept ･ It has advanced scripting capabilities ･ It has a flexible API allowing access to all features of PVSS from an external application”

20. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200720 MDT DCS Alignment On-chamber Off-chamber RasCaM * (5452x) RasLED * (6672x) mask * (6672x) lens * (6672x) RasMuX (768x) Prototype: MiniMasterMuX I 2 C, video, clock, power power JTAG, video, clock Master MuX USA15 MuX (48x) (1 crate) 16 RASNIK (Red Alignment System NIKHEF) NB: slide may not be completely up-to-date

21. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200721 I2C: Inter-IC bus: low-speed (O(1 Mbit/s) ), low-cost Two wires: serial data (SDA) and serial clock (SCL) Devices connected are addressable and all can become master -> like for CAN, protocol supports arbitration http://www.embedded.com/story/OEG20010718S0073

22. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200722

23. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200723 5 compute servers continuously analyze images

24. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200724 Front-End Readout

25. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200725 ATLAS detector Read- Out Drivers ( RODs ) First- level trigger Dedicated links 1600 Read- Out Links TTC First-level trigger accepts cause transfer of data from on-detector buffers to RODs TTC broadcasts LVL1 accepts via tree of optical fibers, can also transmit other messages

26. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200726 TTC protocol LHC bunch crossing time “Biphase mark coding”: a “1” is marked by a transition in the middle of a cell, the clock needed for sampling can be retrieved from the signal received

27. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200727

28. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200728 Example: the MDT front-end readout system http://www.nikhef.nl/pub/experiments/atlas/daq/mrod/mrod.html

29. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200729 1.6 Gbit/s optical fibre, S-Link to ROB CSM link (GOL) 4 - 6 MDT Read- Out Driver (MROD) Chamber 0.8 or 1.6 Gbit/s optical fibre 80 Mbit/s CSM link (GOL) In USA15 24 ch. TDC Chamber Service Module (CSM) up to 18 24 ch. TDC ROL 24 ch. TDC Chamber Service Module (CSM) up to 18 24 ch. TDC 0.8 or 1.6 Gbit/s optical fibre ASICs and FPGAsFPGAs and DSPs MDT front-end readout ~1200 chambers 204 MRODs

30. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200730 On the chambers there is only processing of the data in the TDCs In the TDCs (ASICs) time stamps within a time window corresponding to the maximum drift time in the chambers are extracted and formatted. The TDC data is multiplexed by an FPGA onto a GOL (Gigabit Optical Link). The GOL sender chip is a radiation hard device, developed by CERN. The link uses 8B/10B encoding, which is commonly used for fast serial links: 10 bits are used for encoding 8 bits, so that the resulting signal cannot have a DC component and has enough transitions to recover a clock signal. 8B/10B encoding allows the transmission of "control tokens", i.e. of 10-bit patterns that do not correspond to possible 8-bit patterns. Successive chamber readout cycles (in one readout cycle per TDC one word - if available, otherwise a placeholder word is substituted - is transmitted) are separated by such control tokens on the link between CSM and MROD. Original paper on 8B/10B encoding, IBM Journal of Research, 1983: http://www.research.ibm.com/journal/rd/275/ibmrd2705D.pdf

31. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200731 MRODin SHARC E Memory CSM Memory MRODin Memory CSM Memory MRODin SHARC C Memory CSM Memory SHARC A Event Builder & SLINK interface TIM (TIM module receives TTC information MRODout VME64x ReadOut Link (ROL) VME interface TIM interface RocketIO link FPGA SHARC D SHARC link Bus SHARC link FPGA Bus MROD block diagram

32. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200732 Field Programmable Gate Arrays: two important categories: Static RAM (SRAM) based: reconfigurable: bits in SRAM determine interconnection patterns and in look-up tables (LUTs) logic functions. Complex circuits possible, processors can be embedded in the FPGA, as well as other dedicated parts, e.g. interfaces for high speed links. Usually the "configuration" of the FPGA is stored in on-board flash memory and is loaded into the FPGA at power-up. The functionality of the circuit can be modified by loading a modified configuration into the flash memory. SRAM bits may flip, in particular due to radiation: reloading the configuration is then necessary, unless redundant circuitry and/or special techniques are used Altera, Xilinx Antifuse based: by blowing "antifuses" low resistance links are produced (hence the name antifuse) Only once programmable, but no bit flips in SRAM determining configuration: radiation hard Actel

33. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200733 MRODin: XC2VP7 MRODout: XC2VP20 PowerPC processor blocks not used Xlinix FPGAs used in MROD

34. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200734 Example of complex circuitry that can be implemented in an FPGA: MRODin FPGA

35. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200735 The Xilinx FPGAs used in the MROD are equipped with RocketIO links: bi-directional serial link, transfer speed up to 3.125 Gbit/s, 8 per FPGA 8B/10B coding Used for servicing CSM - MROD links and for communication between FPGAs FPGA technology allows to implement complex and performant dedicated functionality in a single device, modern devices support fast inter-device connects

36. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200736 SHARC DSP ADSP-21160 Digital Signal Processor (SHARC) 512 kByte on-chipSRAM

37. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200737 Handshaking between transmitter and receiver SHARC links

38. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200738 Booting SHARCs: by writing code directly to internal memory (MRODout SHARC, via VME bus) or via link (MRODin SHARCs) using the link interfaces of the MRODout SHARC, which are directly accessible from the VME bus SHARCs communicate via links, MRODout SHARC via VME-bus with crate processor (PowerPC processor running Linux) SHARC software in C++: initialisation of MROD, error detection (using interrupts generated by FPGAs), monitoring (correctness of event data) Direct Memory Access (DMA): data transfer under control of a DMA controller, the CPU only takes care of initializing the controller by supplying it a source address, a destination address and a length of a block to be transferred and then starts the transfer by setting a bit in the controller. Once finished the controller can generate an interrupt if requested to do so, or the end of the transfer can be detected with the help of polling on a bit. The DMA controllers of the SHARC also support "chained transfers": a number of transfers can be done without intervention of the CPU by setting up the necessary information in memory in the form of a linked list and pointing the DMA controller to thead of the list.

39. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200739 S-link is a specification of data link interfaces, where the link transports data mainly in one direction and supports flow control (XON/XOFF mechanism) ATLAS: physical link: bi-directional optical (fiber pair), data throughput (net): 200 MByte/s, 8B/10B coding, CRC error check Successive even fragments are separated by control tokens on the link. Figure from H.C. van der Bij et al., S-LINK: A Prototype of the ATLAS Read-out Link, Presented at the Fourth Workshop on Electronics for LHC Experiments, Rome, 21-25 September 1998) S-link http://hsi.web.cern.ch/HSI/s-link/

40. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200740 VME bus: parallel (up to 64 bit) multi-master asynchronous bus for use in crate. Most ROD crates in ATLAS use VME64x. VME dates back to 1981. The bus is not very fast with respect to present day standards (up to 160 MByte/s), the standardized mechanics are convenient, as well as the vertical cooling and relatively high power dissipation possible For more information: http://www.vita.com/vmefaq.html, http://atlas.web.cern.ch/Atlas/GROUPS/FRONTEND/VMEbus/index.html (access to standards documents only from inside CERN domain)

41. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200741 MROD

42. Jos VermeulenTopical Lectures, Computer Instrumentation, DCS, FE Readout, June 200742 ROL 9U VME crate Crate processor