FF-LYNX Overview FF-LYNX Interfaces FF-LYNX Protocol Fast and Flexible Electrical Links for Data Acquisition and Distribution of Timing, Trigger and Control Signals in Future High Energy Physics Experiments R. Castaldi, G. Magazzù, P. G. Verdini INFN – Sezione di Pisa G. Bianchi, L. Fanucci, S. Saponara, C. Tongiani Dipartimento di Ingegneria dell’Informazione - Università di Pisa Frontier Detectors for Frontier Physics Elba, 24-30 May 2009 Project funded by INFN V Commission Overview FF-LYNX Interfaces Genesis of the FF-LYNX Project • Similar requirements in future High Energy Physics (HEP) experiments on distribution of Timing, Trigger and Control (TTC) signals and Data Acquisition (DAQ) w. r. t. trigger latency, bandwidth, flexibility, radiation hardness, power dissipation and robustness against component failures. • Existing competences at INFN and DII-IET in design of complex digital ASICs, radiation tolerant ICs and communication protocols and interfaces for HEP and space applications. Project phases and targets 1. Analysis of the requirements for data transmission in HEP experiments (CMS Pixel and Strip Detectors, Atlas, ...) and review of existing communication protocols for space (SpaceWire, FiberWire,...) and conventional (Ethernet, FireWire,…) applications. 2. Development of software models of current readout systems, to be used as test bench for next phases. 3. Definition of a custom communication protocol granting the target requirements: Integrated distribution of TTC signals with controlled trigger latency and handling of DAQ processes. Error robustness of critical data (triggers and data frame headers). Flexibility w. r. t. different parameters (data rate, data format, ...) and architectures. Transmission of “hit” data with fixed latency for trigger generation. 4. Validation and performance evaluation of the defined protocol through system and link simulation (SystemC, VHDL, ...). 5. Design of radiation tolerant, low power interfaces that implement the protocol and of data concentrator blocks; design and production of test ICs in standard CMOS technologies (180 nm and below); electrical and radiation tests. 6. Final target: fully characterized IP cores and VHDL models of designed blocks. DAT CLK FF_TX (transmitter) block description TX Buffer (TX_BUF): stores input data received through the data port, awaiting to be transmitted. Frame Builder (FRM_BLD): handles the assembly of data frames. THS Scheduler (THS_SCH): implements the arbitration between triggers and frame headers. Serializer (SER): serializes data to the DAT output stream. FF_RX (receiver) block description Deserializer (DES): splits the DAT input stream into the THS and the FRM channels and deserializes the frame words. THS Detector (THS_DET): detects the arrival of TRG, HDR and SYN sequences in the THS channel. Frame Synchronizer (FRM_SYNC): generates a 40 MHz receiver clock (CLK_40_RX) by synchronizing on the THS channel; CLK_40_RX is then used by an external PLL to reconstruct the transmitter 40 MHz clock (CLK_40). Frame Analyzer (FRM_ANA): examines the fields of the received frames and controls data transfer to the RX Buffer. RX Buffer (RX_BUF): stores received frame words, awaiting to be delivered to the output port. FF-LYNX Protocol Integration of TTC and DAQ transmission in the same packed based protocol, using two physical lines for clock (CLK) and data (DAT): Downlink (towards the detector): clock (master clock recovered in FE electronics from CLK) triggers (highest priority and fixed latency) commands (configuration & monitoring ) Uplink (from the detector): “raw” data (in response to triggers) “trigger” data (to be used for trigger generation) configuration & monitoring data (in response to commands). Link speed: 4xF, 8xF or 16xF (F = master clock frequency = 40MHz in LHC) → 4, 8 or 16 bits transmitted in each master clock cycle, divided in: THS (Trigger, Header, Sync) channel (2 bits) FRM (frame) channel (2, 6 or 14 bits). The THS channel is used to transmit the following 6-bit sequences (bit flip tolerant encoding): TRG, that in the downlink represents triggers while in the uplink is used to mark the beginning of a “trigger” data frame. HDR, that marks the beginning of a “non-trigger” data frame and whose transmission is scheduled when the THS channel is free from TRG transmissions for 3 consecutive master clock cycles. SYN, that is used, together with TRG and HDR sequences, to keep the receiver synchronized with the master clock. The FRM channel is used to transmit data organized in frames variable length frames for “raw” and configuration & monitoring data (lower priority  variable latency) fixed length frames for “trigger” data (highest priority  fixed latency) Frame structure: variable length frames Frame Descriptor (7/12 Hamming encoding), specifying frame length, presence of an optional label and data type Label (16 bits - optional), carrying additional information about the payload, e.g. time stamp associated to “raw” data Payload (0÷15 words, 16 bits each), carrying user data of any format and type (long data packets can be splitted in more consecutive frames) CRC (8 bits - optional), for error protection of label and payload Simulation High-level SystemC and VHDL models of the FF_TX and FF_RX interfaces have been developed and simulated to validate the protocol, including its robustness against transmission errors, and to verify the correct functionality of the interfaces’ architecture Figures of merit have been defined and evaluated through the simulations, e.g. lost trigger rate, data throughput, synchronization recovery time, average packet latency. Frame structure: fixed length frames (“trigger data”) Frame Descriptor (Hamming encoding), specifying the number of hits transmitted in the payload Payload (fixed length), carrying hit data (timing and position) Parity (1 bit), for error protection of payload Trigger data transmission: A hit detected in the FE ASIC triggers the transmission of a fixed-length (n cycles) frame Other hits detected within the n cycles (up to a maximum number defined by the frame length) are also transmitted in the same frame with their relative timing Different protocols can be defined w.r.t. Link Speed (LS), frame length (NC) and maximum Number of Hits (NH) per frame Figures of merit have been defined (e.g. Trigger Transmission Efficiency TTE, Bandwidth Occupancy BO) and high-level C models and Monte Carlo simulation have been used for their evaluation: TRG sent by TX TRG received by RX frame word sent by TX frame word received by RX Possible system architectures A possible uplink data stream (link speed = 8xF  320 Mbps in LHC) Star topology Down-Link: a TX and RX pair between each Front-End chip and a Data Concentrator device, delivering triggers, clock and data/commands to the FE chip. Up-Link: a TX and RX pair between each FE chip and the DC transferring DAQ/command response data; a DCM block inside the DC merges data streams from different sources and optionally performs event building. Ring topology (with redundant links) Triggers propagate along the chain with the highest priority. Each node handles the transmission of data frames generated locally or by previous nodes. Distributed data concentration/event building functions. Redundancy against non-contiguous node failures: 2 selectable, equivalent ports in each RX and TX interface. Protocol versions Version 1: defined and implemented in the interfaces’ models THS/FRM channels Label & CRC 4xF, 8xF or 16xF link speed Version 2: defined but not yet implemented All the version 1 features “Trigger” data transmission A test bench: high-level VHDL model of the CMS Pixel Readout System Output: “Readout file” listing readout data in a simple custom text format. Example: [Event number] [1st Hit Data]...[last Hit Data] [Status byte] ; Input: “Hit file” describing hit pixel addresses and amplitudes in a simple custom text format. Example: [Time index] – [L1T] – [1st Hit Data] ... [last Hit Data] ; where [Hit Data] = ( [ROC num] , [column addr (DC number)] , [row addr] , [hit ampl] ) 1 (0, 14, 52, 5) (1, 11, 9, 5) (1, 18, 112, 5) (2, 4, 146, 5) (2, 7, 129, 5) (2, 10, 147, 5) 00000000 ; 2 (0, 11, 151, 5) 00000000 ; 3 (0, 4, 116, 5) (1, 6, 85, 3) (1, 15, 73, 3) (1, 22, 47, 3) (2, 16, 95, 3) 00000000 ; 4 (0, 3, 75, 5) (0, 24, 26, 3) (1, 5, 47, 3) (1, 14, 35, 3) (1, 16, 117, 3) (1, 18, 79, 3) (2, 9, 98, 3) 00000000 ; 5 (0, 19, 47, 5) (2, 17, 72, 3) (2, 20, 59, 3) 00000000 ; 11 - 0 - (0, 12, 01, 3) (2, 20, 140, 3); 30 - 0 - (1, 3, 158, 4) (1, 15, 59, 3); 40 - 0 - (0, 8, 152, 6) (0, 17, 18, 5) (2, 6, 47, 3); 50 - T - (0, 14, 52, 5) (1, 11, 9, 5) (1, 18, 112, 5) (2, 7, 129, 5) (2, 4, 146, 5) (2, 10, 147, 5); 63 - 0 - (0, 2, 15, 5) (1, 21, 123, 3) (2, 24, 99, 3); 70 - T - (0, 11, 151, 5) (2, 4, 14, 3); 80 - T - (0, 4, 116, 5) (1, 6, 85, 3) (1, 15, 73, 3) (1, 22, 47, 3) (2, 16, 95, 3); 90 - 0 - (2, 2, 147, 5); 100 - 0 - (1, 23, 114, 5) (1, 15, 78, 3) (2, 17, 55, 3); 110 - T - (0, 3, 75, 5) (0, 24, 26, 3) (1, 16, 117, 3) (1, 14, 35, 3) (1, 18, 79, 3) (1, 5, 47, 3) (2, 9, 98, 3); 130 - T - (0, 19, 47, 5) (2, 20, 59, 3) (2, 17, 72, 3); 204 - 0 - (0, 16, 46, 5); Functional simulations: VHDL models of FF-LYNX interfaces will be included in the CMS Pixel readout system model for an overall functionality verification and performance evaluation. Importance of high level system simulations as a first, fundamental verification step of the design flow. Trigger validation of hits and DC readout mode setting Hit receiving by PUCs and column drain in a DC ROCs’ readout cycle starting by TBM ROCs’ readout cycle starting by TBM