Backend Control Systems in uTCA HCAL Requirements January 21, 2010 Jeremiah Mans

January 21, 2010 HCAL Requirements for SLHC Controls 2 Outline Architectural concept for backend upgrade Requirements for a control system R&D work carried out to date

January 21, 2010 HCAL Requirements for SLHC Controls 3 Strawman Architecture uHTR Control (MCH site) uHTR Control MCH (commercial) DAQ + Timing (DTC) MCH Site(s) (1-2 special hub slots with connectivity to all other slots) Links/ uTCA HTR from FE Trigger Links (Protocol/connectors/density to be defined)

January 21, 2010 HCAL Requirements for SLHC Controls 4 uTCA HTR Block Diagram Virtex 6 (XC6VLX240T) 12 link pairs System Microcontroller Enet Switch I 2 C System Control Bus Configuration, Local DAQ SNAP12 12-way optical receiver SNAP12 12-way optical receiver Daughter card (10 Gbps) or direct-drive (6.5 Gpbs) Commercial MCH Global DAQ LHC Clock Fast Controls DTC

January 21, 2010 HCAL Requirements for SLHC Controls 5 Observations from these strawmen Two control paths defined by the standard – I2C (+IPMI) : core required infrastructure for all card functionality, low data rate – Ethernet : gigabit on Port 0/Port 1 is extremely common and provides the dominant control path for high bandwidth purposes I2C is extremely low data rate, appropriate for functionalities such as temperature monitoring, triggering FPGA reconfiguration, but not DAQ or loading FLASH memories. Ethernet will carry the load for all configuration processes

January 21, 2010 HCAL Requirements for SLHC Controls 6 HCAL Requirements FPGA reprogramability in all situations – Even if the FPGA is misconfigured or unconfigured, it should be possible to replace the FLASH program without removing the card from the system => direct Ethernet access to the microcontroller High-rate local DAQ capability – The local DAQ should be capable of at least 10 Hz readout of unsuppressed data (to support testbeam, calibration, and other key operational/performance goals) => direct Ethernet access to the FPGA Low latency of configuration – The configuration of the many LUTs and control registers required by the system should be possible to complete within 30 seconds (less is desirable)

January 21, 2010 HCAL Requirements for SLHC Controls 7 HCAL Requirements (2) Communication with hardware should not require special kernel-level drivers – Implies use of standards such as UDP rather than raw Ethernet A standard CMS-agreed IP packet format and PC-side HAL should be used – This standard should be compatible with both FPGAs and microcontrollers – At least one implementation (on the FPGA side) of the format based on a simple (internal) parallel bus should be available – Standard techniques (e.g. internal addresses to read) should be defined to allow the identification of the card type, the chip on the card, and the firmware version

January 21, 2010 HCAL Requirements for SLHC Controls 8 HCAL Requirements (3) Access to hardware should have minimal requirements – In current system, commandline debugging is possible even during operations. This is crucial during development and very helpful during operations. – In the UDP/IP space, it should be simple to determine the port and IP addresses. (e.g. fixed port and crate/slot/chip IP addresses or DNS names) – Also implies allowing access from multiple processes/computers

January 21, 2010 HCAL Requirements for SLHC Controls 9 HCAL Requirements (4) Support for forbidding multiple access (from multiple computers or processes) would be valuable – Protocol cannot handle every case, but having a 'bus lock' concept [locking access to a specific IP+port] would probably be valuable. Leads to questions about timeouts and recovery of resources! Short timescale – HCAL will begin tests (including beam tests) with uTCA hardware during this year. We would prefer to work with a standard from the beginning, but we realize that any such standard would likely be incomplete and preliminary in 2010

January 21, 2010 HCAL Requirements for SLHC Controls 10 Open Questions What level of software integration between the I2C/IPMI and the Ethernet control systems is appropriate?

January 21, 2010 HCAL Requirements for SLHC Controls 11 R&D carried out Ongoing R&D requires a temporary solution for the time being. – Also provided a chance to think through requirements for this meeting in more detail! Virtual parallel bus – Protocol defines a 'virtual parallel bus' on the FPGA side (32-bit data, 32-bit word addressing (2 34 bytes) – Uses a subset of the control signals of VME (STROBE, WRITE, DTACK, BERR) Rationale – Parallel bus (VME-like) is a commonly-understood paradigm for many engineers – Interface is both simple and powerful [should be capable of ~32 M transactions/sec (~100 MBps)]

January 21, 2010 HCAL Requirements for SLHC Controls 12 Wire-Level Protocol Uses UDP transport Protocol-specific header to identify the packet (for use on the software end) Each packet contains one or more bus transactions (READ, WRITE, RMW) – READ and WRITE can be multiword actions (each read/write is separately generated on the internal local bus) – RMW is defined as A <= ( A & CONST1) | CONST2 Software is responsible for splitting bus transactions to keep both incoming and outgoing packets below the MTU (assuming the Ethernet MTU and standard overheads)

January 21, 2010 HCAL Requirements for SLHC Controls 13 Status Current firmware: – Implements ARP + ICMP-ping in a miniCTR1 (Virtex 5 XC5V30 class) using 1% of resources (except 2% of block RAMs) – Bus transaction engine written, still need to glue together with the other packet handlers and add checksum calculation, etc Could have a ~working firmware set by mid February Missing: – Any standards agreement! – Bus locking – Standards on card identification – Robust HAL supporting overlapped operations

January 21, 2010 HCAL Requirements for SLHC Controls 14