GBT-SCA Slow Control Adapter ASIC LHCb Upgrade Electronics meeting 12 June 2014

Design Team Core Digital Logic: Christian Paillard, Alessandro Caratelli e-port interface: Sandro Bonacini ADC: icSparkling IP block DAC: Xavier Llopart (Medipix IP block) Chip Assembly: Christian Palliard, Rui F. Oliveira AOB: Kostas Kloukinas, Paolo Moreira 12 June 14 Alessandro.Caratelli@cern.ch

SCA in the GBT system 12 June 14 Alessandro.Caratelli@cern.ch

The GBT System A Single Link for: Readout (DAQ) Embedded Electronics Control Room Optical Link A Single Link for: Readout (DAQ) High speed unidirectional (up-link) Trigger data (up-link) Timing Trigger and Control (TTC) Clock reference and synchronous control (down-link) Trigger decisions and control (down-link) Low and fixed latency Experiment control (SC/DCS/ECS) Modest bandwidth (bidirectional link) Custom ASICs in the detectors: Radiation Tolerant: Total dose & Single Event Upsets Commercial components in the control room FPGAs used to implement multi-way transceivers The communication between the front-end electronics embedded in the detectors and the control room is manly composed of three functional subsystems • A fast timing distribution system responsible to deliver to the experiment the system clock and several fast trigger signals and some fast signals from the detector to the control room, • A data acquisition link carrying the collected data from the detector to the control room, • A slow control system carrying bidirectional traffic from and to the control room and the embedded electronics in the detector. The GBT system provides a single bidirectional link for carrng the thre types of traffic. An appropriate bandwidth is allocated to each of the three subsystems. 12 June 14 Alessandro.Caratelli@cern.ch

GBT-SCA front-end interconnects e-port GBTX GBT on FPGA Slow Control DAQ Timing and Trigger GBT-SCA FE ASICs Embedded electronics Control room User busses and I/O for Control and monitoring Data Timing ASIC dedicated to slow control functions. System Upgrades for SLHC detectors. Replacement for the CCU & DCU ASICs CCU: Communication Control Unit DCU: Detector Control Unit in CMS). Flexible to match the needs of different Front-End systems. Technology: CMOS 130nm using radiation tolerant techniques. The slow control system is the one uncharged to distribute control sequences and collect status information from the embedded peripheral electronics in the detector. Such as front-end ASICs or monitoring chips for control of environmental variables or any other specific module which needs to be remotely controlled the SCA chip is the embedded node of the system which is responsible to translate the unified packets sent by the control room and redirect to the selected peripheral through one of the physical ports. It connects to a dedicated e-link port on the GBTX ASIC, with a fixed bandwidth of 80MHz. The GBT communication is totally transparent to the slow control protocol. 12 June 14 Alessandro.Caratelli@cern.ch

The Slow Control Adapter ASIC Dual redundant e-Ports for GBTX e-links. Network Controller Arbiter 16 I2C masters (7bit/10bit addressing) 1 JTAG master controller 1 SPI master (with 8 slave select) 32 General purpose digital I/O lines individually programmable as input / output / interrupt input 31 Analog inputs multiplexed in a 1 bit single slope integrative ADC 4 independent analog output (8 bits resolution) Auxiliary I2C port for testability The remote devices controlled by the SCAs are seen from the control computer as remote independent channels, each one with a particular set of control registers and/or allocated memory locations. \ The channels operate independently from each other in order to allow concurrent transactions and perform concurrent transfers to their end-devices. The NC is encharged to collect the requests from the control room, interpetate it and redirect to a specific channel interface, as well collect the replyes from the interfaces when they request attention through an internal interrupt system and resend to the control room. Figure 1. The SCA top level block diagram 12 June 14 Alessandro.Caratelli@cern.ch

Communication architecture 12 June 14 Alessandro.Caratelli@cern.ch

Communication Architecture The Three Communication Layers of the SCA: GBT-link Layer Connects the GBT ASIC to the Control Room electronics via a point-to-point, bi-directional, 4.8Gbps, optical link using a special, SEU robust, transport protocol. The transport protocol is totally transparent to the user data. e-link Layer Connects to the SCA ASIC via a point-to-point, bi-directional, 80Mbps link, using a packet oriented transport protocol (HDLC protocol). Data packets are encapsulated by the GBT-link protocol. Channel Layer Connects to the Front-End ASICs via bi-directional links. SCA on-chip peripherals are also Channel Blocks. Channel Blocks are controlled via a command driven protocol. The Channel Protocol Commands are encapsulated by the e-link protocol. 12 June 14 Alessandro.Caratelli@cern.ch

GBT-link Packet Format Fixed packet length: 120bits Packet transmission rate: 1/25ns Data transmission rate: 4.8 Gbps Fixed bandwidth allocation: Trigger path: 640 Mbps Control path: 160 Mbps 1 internal e-link (for GBT management) 1 external e-link (for GBT-SCA chip) 40 MHz DDR (80 Mbps) Data path: 2.56 Gbps 10 e-links @ 320 Mbps 20 e-links @ 160 Mbps 40 e-links @ 80 Mbps Data flow: Symmetrical, Bi-directional data transmission. Transmission of GBT-packets is continuous. Data from e-link ports are muxed/demuxed in the GBT-link stream. GBT data path is unaware of the e-link transfer protocol. 2 bits H IC EC FEC 80 bits 32 bits 4 bits Neverless it is possible to connect up to 40 SCA ASICS to 1 GBTx module, on the data path, in case the user wants to have an optical link dedicated for slow control purposes. 12 June 14 Alessandro.Caratelli@cern.ch

HDLC Packet Format HDLC Packet: 8 bits SOF 01111110 Address Control DATA FCS EOF 16 bits 16∙n bits 6 bits HDLC packet structure: HDLC Packet: SOF/EOF: Frame delimiter character. Address: Packet destination address. Control: Indicates the type of the data packet. FCS: Frame Check Sequence for transmission error detection. (𝐺(𝑥) = 𝑥 16 + 𝑥 12 + 𝑥 5 + 1) Uses bit-stuffing techniques in order to achieve symbol synchronization. The frame delimiter contains 6 consecutive ones; Any sequence of 5 ones in the stream is stuffed with one following zero at the transmitter side Any sequence of more than 6 ones is considered to be frame abort or channel idle signalling Bi-directional operation. Each received command packet get acknowledged or rejected by the SCA 12 June 14 Alessandro.Caratelli@cern.ch

SCA Packet Formats SCA Packet: HDLC packet : 8 bits SOF Address Control DATA FCS EOF 16 bits 16∙n bits 6 bits HDLC packet : 8 bits 16∙n bits CH# TR# CMD Data LEN SCA up-link packet structure: ERR SCA down-link packet structure: SCA Packet: CH#: Specifies the SCA Channel interface to be addressed TR#: Incremental identification number for each transmitted command. CMD: Is a command code that specifies the request. The operation can refer to a specific internal register of the channel or a front-end ASIC destination address. In this case an address field follows the command. ERR: In case of an error has encountered, return an error otherwise returns 0x00. LEN: Is a field that specifies the DATA field length. DATA: Is an optional variable length field upon LEN value. 12 June 14 Alessandro.Caratelli@cern.ch

Channel Commands At the reception of any request packet, the SCA replies with an acknowledge packet The acknowledge packet include information of the status of the operation and eventually the data value required. Each command can be directed to a specific channel interface. Upon reception of a command, the channel performs the required operation on its interface. The command field and the data content of a given packet is interpreted differently according to the channel to which it is addressed. Each channel type has a specific set of valid commands. Channels receiving an invalid command do not execute any action and the error condition is reported to the user. Enable/Disable command for each channel When not enabled, the interface core is kept in Power Down Mode to reduce power consumption. At the enable, the channel interface get reset. 12 June 14 Alessandro.Caratelli@cern.ch

Implementation 12 June 14 Alessandro.Caratelli@cern.ch

e-link & e-port Block Diagram SCA Network Controller RX FIFO RX control PHY RXCLK RXDATA ATLANTIC Bus TX control TXDATA TX FIFO Clk 40 MHz RESET e-port implementation for the GBT-SCA e-link interface: electrical, bi-directional chip-to-chip interconnect at 80Mbps. HDLC (High Level Data Link Control) packet oriented communication protocol. Special commands: CONNECT, RESET, TEST (loopback) Acknowledge transactions using the HDLC protocol mechanism (packet numbering). 12 June 14 Alessandro.Caratelli@cern.ch

Dual Redundant e-ports The GBT-SCA is equipped with a Primary and a Secondary e-port to be connected with two adjacent GBTX chips for redundancy Only one e-port is needed for operation and can be active at a time Data from the inactive e-port is discarded Switching between ports is command driven. (HDLC “CONNECT” command) Clock activity or noise on the differential input of the inactive link is discarded Upon start up, the user application software should send a “CONNECT” command to activate the Primary e-port from the Primary e-link. To switch to the Secondary e-port a “CONNECT” command should be send through the Secondary e-link. Figure 2. GBT-SCA to GBTX connectivity Figure 3. Logical view of HDLC e-ports 14/05/2014 GBT-SCA Design review

Physical Layer (electrical) SLVS (Scalable Low Voltage Standard) JEDEC standard: JESD8-13 Differential voltage based signaling protocol. Voltage levels compatible with deep submicron processes. Low Power, Low EMI Line impedance: 100 Ohm Differential mode: 400 mV Common mode: 0.2V 0.2V V 200mV 12 June 14 Alessandro.Caratelli@cern.ch

Auxiliary I2C port Auxiliary I2C port for testability Allows direct access to Network Controller by-passing the e-ports. Receive SCA commands and returns replies Hardwired “Test Enable” pin Possibility to use as an expansion port 14/05/2014 GBT-SCA Design review

The SPI channel Synchronous bit oriented serial link. Full duplex synchronous serial data transfer Configurable single transaction length up to 128 bits RW Configurable communication speed from 156KHz to 20MHz settable with 128 possible values Asynchronous reset signal with configurable pulse length Acknowledge packet to report the end of the communication Supported working modes: (0,0) (0,1) (1,0) (1,1). Possibility to invert the logical level of the output signals during the inactivity time. Figure 5. The SPI channel scheme 14/05/2014 GBT-SCA Design review

The JTAG channel 128 bit shift registers to hold the TDO/TDI and TMS bit-streams Protocol implemented in software Configurable single transaction length up to 128 bits RW Configurable communication speed from 156KHz to 20MHz settable with 128 possible values Asynchronous reset signal with configurable pulse length Acknowledge packet to report the end of the communication Possibility to invert the TCK edge synchronously to which the transmitted bits on the TDO and TMS lines are latched Possibility to invert the TCK edge synchronously to which the received bits on the TDI line are evaluated. Possibility to invert the logical level of the output signals during the inactivity time. Figure 6. The JTAG channel scheme 14/05/2014 GBT-SCA Design review

The I2C channels 16 independent I2C master channels Concurrent operation of all 16 channels Support is limited to single bus master topologies. 7-bit and 10-bit addressing modes. Single-byte and Multi-byte transfer modes. Read-Modify-Write mode that allows for AND, OR, XOR logic operations with a Mask register. Programmable transfer rate from 100Kbps to 1Mbps. Power Down mode per channel Figure 4. I2C channel block diagram 14/05/2014 GBT-SCA Design review

The GPIO channel 32 general-purpose I/O signals. All general-purpose I/O signals are bi-directional having individually programmable direction. All general-purpose I/O signals can be three-stated. General-purpose I/O signals programmed as inputs can cause interrupt requests. General-purpose I/O signals programmed as inputs can be registered at raising edge of system clock or at user programmed edge of external clock. Alternative input reference clock signal from external interface. Power Down mode Figure 7 - The GPIO channel 14/05/2014 GBT-SCA Design review

The DAC channel 4 independent Digital to analog converters. The single DAC is an IP Block from MEDIPIX team Voltage output: 0.0V – 1.0V 8 bit resolution Figure 8 – The DAC channel 14/05/2014 GBT-SCA Design review

ADC channel DIGITAL ANALOG ADC Control logic Calibration logic 12-bit ADC (LSB = 244 µV) Single -slope architecture Input range: GND < Vin < 1 V Single supply voltage: VDD = 1.5 V System clock frequency: 40 MHz Max. conversion rate ≈ 3.5 KHz Max conversion time ≈ 600 μsec Operating temperature: -30°C to +80°C 32 Multiplexed Input channels 1 channel occupied by an on chip Temp sensor All channels feature a switchable 10 μΑ current source to support Pt sensors. Power consumption 350uA @ 1.2V (420uW analog part ) Power Down mode Automatic Offset cancelation & Gain correction Pre-calibration at production testing phase DIGITAL ANALOG ADC Control logic Calibration logic E-fuses read logic E-fuses bank Supervising logic + Wishbone interface Figure 9. The ADC channel scheme 14/05/2014 GBT-SCA Design review

Layout E-Fuses DAC Prompt ADC analog ADC calib. logic JTAG channel SPI channel I2C channels Network controller GPIO channel E-Link 3.6 x 3.6 mm 12 June 14 Alessandro.Caratelli@cern.ch

Packaging (draft) Package Type: LFBGA Low Profile Fine Pitch BGA (Chip Scale Package) Ball Pitch: 0.8 mm Size: 12 x 12 mm Height: 1.2-1.7 mm Pin count: 196 Preliminary pinout Manufacturers: ASE via IMEC NOVAPACK 12 June 14 Alessandro.Caratelli@cern.ch

Final Verifications Functional simulation at the top level Post Layout simulations Simulation with SEU injection Ultrasim Mixed-Signal simulations SCA emulation on FPGA Protocol tests with commercial devices 12 June 14 Alessandro.Caratelli@cern.ch

FPGA test bench 1/2 Software FPGA test firmware SCA logic 14/05/2014 I2C Slave x4 USB control logic SCA packet gener. E-Port Master Parallel I/O ctrl SPI Slave Multimode Comm. control JTAG slave Transf. control flags SCA send packet SCA rec packet GPIO data SPI data JTAG data Slaves Configure Test System config. I2C0 data I2Cn data Drivers + USB bluster + interface etc. Low level func. to handle GBT-SCA communication Low level func. to handle slaves Testing software: Start random transaction on random channels Whait with time-out replies from the SCA logic Compare data received on slave interfaces with the expected behaviour respect sent requests Write operation log and error log Test passed when no errors are encountered Software FPGA test firmware 14/05/2014 GBT-SCA Design review

JTAG test with commercial hardware SCA logic I2C Slave x4 USB control logic SCA packet gener. E-Port Master Parallel I/O ctrl SPI Slave Multimode Comm. control Transf. control flags SCA send packet SCA rec packet GPIO data SPI data JTAG data Slaves Configure Test System config. I2C0 data I2Cn data Drivers + USB bluster + interface etc. Low level func. to handle GBT-SCA communication Low level func. to handle slaves Program en external commercial FPGA through the GBT-SCA emulated on an other FPGA: Read FPGA configuration file (ISE output file containing series of directives for JTAG interface to program the target Xilinx FPGA) Find the target FPGA in the JTAG chain Generate corresponding commands in SCA format Send the commands to the GBT-SCA emulated in the main FPGA Write error log of unsuccessful operations Software FPGA test firmware I2C interface SPI interface JTAG interface 14/05/2014 GBT-SCA Design review

Command line interface FPGA test bench 2/2 FPGA test firmware Nios II Processor Handle shell interface E-port drivers Slaves drivers Handle communication with SCA Real-time verification of concurrent operations performed by the SCA Report Operation information and identified errors Comm. control E-Port Master SCA logic Comm. control E-Port Master I2C Slave x4 I2C Slave x4 I2C Slave x4 I2C Slave x4 Drivers and USB bluster Command line interface JTAG port Parallel I/O ctrl Timer banks SPI Slave Multimode SDRAM controller JTAG slave PLL 12 June 14 Alessandro.Caratelli@cern.ch

Specifications Documentation: “Users Manual” document (draft) “Table of commands” 12 June 14 Alessandro.Caratelli@cern.ch

Status & Planning Design Readiness review on 14/5/2014 Design Status Suggestions for design verifications Minor design modifications (suppression of external biasing resistor) Design Status Ready for submission Participate on the GBT-chipset engineering run scheduled for this month (June 2014) Estimated TAT: 4-5 months Yield a few hundred SCA chips Test Bench Preparation 12 June 14 Alessandro.Caratelli@cern.ch

Thank you 12 June 14 Alessandro.Caratelli@cern.ch