MicroTCA A look at different options... My personal opinion from a CMS Trigger perspective Greg Iles 12 October 2010

Greg Iles, Imperial College Considerations MicroTCA – Very Flexible Mixed blessing Fixed decisions Double width card is a nice size 149mm x 183.5mm Space for CPU/FPGA + RAM + PSU Almost fixed decisions..... Number of slots: 12 UDP v TCP: Not discussed in this talk RTM: Not discussed in this talk Mid or Full size? Connectivity? Cooling? Height Width Which crate shall I buy? 6 July 2010 Greg Iles, Imperial College

Size – Part A: Component Height Full-Size: Inside = 22.85 Env = 22.45 Mid-Size: Inside = 12.86 Env = 13.28 / 11.85 / 10.85 Notch = 13.61 FMC = 12.4mm (9.5/8.0 2.9/4.4) SFP+ = 9.1 Avago POD = 12.9 (On the limit, Inc heat sink) Reflex SNAP12 = 11.0 (no heat sink), 15.0 QFSP = 9.7 (no heat sink), 16.0 or 23.0 BGA = 3.5 => 7.55 for heatsink (MS Env2) Not a problem in the future? Is it pluggable? Requires reliable optics... All dimensions in mm 6 July 2010 Greg Iles, Imperial College

Size – Part B: Heat Dissipation Commercial Card Mid & Full Size Typical: 25-45W Worst: 32-50W Large FPGA - 48 links at 10G Large amount of processing 20W - 30W? Air temp 20 deg C Max temp 60 deg C If 20W => Require < 2 C/W External Width:40 (42) Height:7.5 (19..5) Length:57 (73) Mid Full FT/MIN M/S C/W C/W 200 1 4.5 1.9 300 1.5 3.4 1.5 400 2 2.9 1.3 500 2.5 2.6 1.2 600 3 2.3 1.1 700 3.5 2.2 1.0 800 4 2.0 0.9 6 July 2010 Greg Iles, Imperial College

Connectivity: The basics Tongue 1 Fabric A (GbE) + IPMI Tongue 2 Multiple options: Fabric B (SATA) 12 slots + 2 clocks Fabric B (SATA) 6 slots + 3 clocks 4 clocks Tongues 3&4 Fabric DEFG (PCIe, SRIO, 10GbE) Important Differences 6 July 2010 Greg Iles, Imperial College

Greg Iles, Imperial College Connectivity: Redundant Telecom – Dual Star DAQ 6.4Gb/s GbE T1 Fast Control NOT SerDes 400Mb/s TTC/TTS SATA or SAS T2 FatPipe x4 lanes e.g. PCIe SRIO T3-T4 Custom e.g. Switch T2 LHC 40MHz Clk Clk Out T2 Clk In Not shown: 8 spare ports 2 spare clks 6 July 2010 Greg Iles, Imperial College

Greg Iles, Imperial College Connectivity: Physics Profile 6 July 2010 Greg Iles, Imperial College

Greg Iles, Imperial College Towards a CMS system DTC by Eric Hazen, Boston University Purpose Distribute Clock Distribute Fast Control Receive Fast Feedback DAQ funnel Fixed Latency, NOT Serdes, 400Mb/s Prototype built on MCH from NAT, but not required. Vendor independent 6 July 2010 Greg Iles, Imperial College

Greg Iles, Imperial College Decisions Decided to use MCH2 as AMC13 i.e. Crate NOT used in redundant mode Redundant system could be built, but it would lock experiments into a particular MCH vendor that would supply Tongue 1 Route TTC/TTS on port 3 Decided to use LVDS. Simple, fixed latency and low power (ideal for tongue 2) Route DAQ on port 1 Port 1 allows protocol agnostic switch on ports 8-11 6 July 2010 Greg Iles, Imperial College

Dual Star, Telecom Clocks MCH2: LHC-CLK, TTC & TTS and DAQ Concentrator MCH2 or AMC13 12 full size AMC slots MCH1 MCH1 providing GbE and standard functionality Vadatech VT891 6 July 2010 Greg Iles, Imperial College

Greg Iles, Imperial College Pro: JSM Slot possible in all crates 12 Mid Size AMC slots Pro: Basic design more widely available.. Pro: Allows RTM Schroff Part 11850-015 6 July 2010 Greg Iles, Imperial College

Greg Iles, Imperial College Cooling By default front to back cooling Failure does not lead to water dripping over electronics Easy access for maintenance of cooling units Can mix and match computers, crates and opto patch panels in same rack Cannot double stack a rack unless designed for it We need flexibility Request hardware modification kit to convert from front/back to vertical 6 July 2010 Greg Iles, Imperial College

Greg Iles, Imperial College Conclusions Mid or Full size? Mid Size would make us compatible with the mechanics of multiple vendors Very little room for heatsinks Connectivity? Redundant Telecom crate with port 2/3 routed to MCH allows us to separate DAQ and TTC/TTS into dedicated channels (SerDes CML & fixed latency LVDS) Cooling? Front-2-Back cooling probably better, but infrastructure is vertical Require crates to be capable of both 6 July 2010 Greg Iles, Imperial College

Questions ? DRAFT document on MicroTCA in physics: http://giles.web.cern.ch/giles/projects/slhc/The_CMS_uTCA_Crate_v0.6.pdf

Greg Iles, Imperial College Communication PHY Protocol format for register read/write capability over large latency communication medium i.e GbE in this case Single data packet, multiple transactions UDP/IP can be implemented in VHDL Two versions already exist TCP/IP usually implemented with processor PowerPC hardcore MicroBlaze soft-core If hardware accelerated > 500Mb/s EMAC UDP, or TCP Transaction Engine I2C Core GTX Core DAQ Core 6 July 2010 Greg Iles, Imperial College

Software: Architecture User code User code User code Software: Architecture Control LAN Fabric Hardware controller PC separates the Control LAN and the User code from the Hardware LAN and the devices Unlike current TS architecture, all network traffic hidden from end user Made possible by common interface layer within the firmware and mirrored within the software Single Multicore host Network Interface Multiplexer layer Transport Adapter Hardware LAN Fabric Kernel Async. IO services 6 July 2010 Greg Iles, Imperial College

Rear Transition Module Physics xTCA working group Interested parties seem to be DESY and Schroff 6 July 2010 Greg Iles, Imperial College

Greg Iles, Imperial College 6 July 2010 Greg Iles, Imperial College