1. MicroTCA A look at different options...
My personal opinion from a CMS Trigger perspective
Greg Iles
12 October 2010

2. Greg Iles, Imperial College
Considerations
MicroTCA – Very Flexible
Mixed blessing
Fixed decisions
Double width card is a nice size
149mm x 183.5mm
Space for CPU/FPGA + RAM + PSU
Almost fixed decisions.....
Number of slots: 12
UDP v TCP: Not discussed in this talk
RTM: Not discussed in this talk
Mid or Full size?
Connectivity?
Cooling?
Height
Width
Which crate shall I buy?
6 July 2010
Greg Iles, Imperial College

3. Size – Part A: Component Height
Full-Size:
Inside = 22.85
Env = 22.45
Mid-Size:
Inside = 12.86
Env = / / 10.85
Notch = 13.61
FMC = 12.4mm (9.5/ /4.4)
SFP+ = 9.1
Avago POD = 12.9 (On the limit, Inc heat sink)
Reflex SNAP12 = 11.0 (no heat sink), 15.0
QFSP = 9.7 (no heat sink), 16.0 or 23.0
BGA = 3.5 => 7.55 for heatsink (MS Env2)
Not a problem in the future?
Is it pluggable?
Requires reliable optics...
All dimensions in mm
6 July 2010
Greg Iles, Imperial College

4. Size – Part B: Heat Dissipation
Commercial Card
Mid & Full Size
Typical: 25-45W
Worst: 32-50W
Large FPGA - 48 links at 10G
Large amount of processing
20W - 30W?
Air temp 20 deg C
Max temp 60 deg C
If 20W => Require < 2 C/W
External Width:40 (42)
Height:7.5 (19..5)
Length:57 (73)
Mid
Full
FT/MIN M/S C/W C/W
6 July 2010
Greg Iles, Imperial College

5. Connectivity: The basics
Tongue 1
Fabric A (GbE) + IPMI
Tongue 2
Multiple options:
Fabric B (SATA) 12 slots + 2 clocks
Fabric B (SATA) 6 slots + 3 clocks
4 clocks
Tongues 3&4
Fabric DEFG (PCIe, SRIO, 10GbE)
Important Differences
6 July 2010
Greg Iles, Imperial College

6. Greg Iles, Imperial College
Connectivity: Redundant Telecom – Dual Star
DAQ
6.4Gb/s
GbE T1
Fast Control
NOT SerDes
400Mb/s
TTC/TTS
SATA
or SAS T2
FatPipe
x4 lanes
e.g.
PCIe
SRIO
T3-T4
Custom
e.g. Switch T2
LHC 40MHz Clk
Clk Out T2
Clk In
Not shown: 8 spare ports 2 spare clks
6 July 2010
Greg Iles, Imperial College

7. Greg Iles, Imperial College
Connectivity: Physics Profile
6 July 2010
Greg Iles, Imperial College

8. Greg Iles, Imperial College
Towards a CMS system
DTC by Eric Hazen,
Boston University
Purpose
Distribute Clock
Distribute Fast Control
Receive Fast Feedback
DAQ funnel
Fixed Latency, NOT Serdes, 400Mb/s
Prototype built on MCH from NAT, but not required.
Vendor independent
6 July 2010
Greg Iles, Imperial College

9. Greg Iles, Imperial College
Decisions
Decided to use MCH2 as AMC13
i.e. Crate NOT used in redundant mode
Redundant system could be built, but it would lock experiments into a particular MCH vendor that would supply Tongue 1
Route TTC/TTS on port 3
Decided to use LVDS. Simple, fixed latency and low power (ideal for tongue 2)
Route DAQ on port 1
Port 1 allows protocol agnostic switch on ports 8-11
6 July 2010
Greg Iles, Imperial College

10. Dual Star, Telecom Clocks
MCH2: LHC-CLK, TTC & TTS and DAQ Concentrator
MCH2 or
AMC13
12 full size AMC slots
MCH1
MCH1 providing GbE and standard functionality
Vadatech VT891
6 July 2010
Greg Iles, Imperial College

11. Greg Iles, Imperial College
Pro: JSM Slot possible in all crates
12 Mid Size AMC slots
Pro: Basic design more widely available..
Pro: Allows RTM
Schroff Part
6 July 2010
Greg Iles, Imperial College

12. Greg Iles, Imperial College
Cooling
By default front to back cooling
Failure does not lead to water dripping over electronics
Easy access for maintenance of cooling units
Can mix and match computers, crates and opto patch panels in same rack
Cannot double stack a rack unless designed for it
We need flexibility
Request hardware modification kit to convert from front/back to vertical
6 July 2010
Greg Iles, Imperial College

13. Greg Iles, Imperial College
Conclusions
Mid or Full size?
Mid Size would make us compatible with the mechanics of multiple vendors
Very little room for heatsinks
Connectivity?
Redundant Telecom crate with port 2/3 routed to MCH allows us to separate DAQ and TTC/TTS into dedicated channels (SerDes CML & fixed latency LVDS)
Cooling?
Front-2-Back cooling probably better, but infrastructure is vertical
Require crates to be capable of both
6 July 2010
Greg Iles, Imperial College

14. Questions ? DRAFT document on MicroTCA in physics:

15. Greg Iles, Imperial College
Communication
PHY
Protocol format for register read/write capability over large latency communication medium
i.e GbE in this case
Single data packet, multiple transactions
UDP/IP can be implemented in VHDL
Two versions already exist
TCP/IP usually implemented with processor
PowerPC hardcore
MicroBlaze soft-core
If hardware accelerated > 500Mb/s
EMAC
UDP, or
TCP
Transaction
Engine
I2C
Core
GTX
Core
DAQ
Core
6 July 2010
Greg Iles, Imperial College

16. Software: Architecture
User code
User code
User code
Software: Architecture
Control LAN Fabric
Hardware controller PC separates the Control LAN and the User code from the Hardware LAN and the devices
Unlike current TS architecture, all network traffic hidden from end user
Made possible by common interface layer within the firmware and mirrored within the software
Single Multicore host
Network Interface
Multiplexer layer
Transport Adapter
Hardware LAN Fabric
Kernel Async. IO services
6 July 2010
Greg Iles, Imperial College

17. Rear Transition Module
Physics xTCA working group
Interested parties seem to be DESY and Schroff
6 July 2010
Greg Iles, Imperial College

18. Greg Iles, Imperial College
6 July 2010
Greg Iles, Imperial College