RT2010 & xTCA workshop a (personal) summary Markus Joos CERN-PH/ESE

xTCA ATCA uTCA / AMC General A success in the telecom industry (market volume ~109 $) 80% of the boards use GBE in the fabric network (15% PCIe) Workshop had a focus on control (IPMI, S/W) aspects AXIe (http://www.axiestandard.org) uTCA / AMC Mid-size (4HP) most popular AMC height Nothing revolutionary new General Little information on real-life experience with HA solutions (technology still too young?) Performance No much about actual transfer speeds and latencies of e.g. GBE or PCIe at system level Recommendation for PCIe: Avoid “reads” (use remote write)

uTCA / AMC for physics RTMs defined ADF/ZD connector, 20 (compact AMC) or 30 LVDS pairs RTM ~same size as AMC RTM gets 12 V (+3.3 V) from AMC Power: AMC + RTM = 80W (recommended: 50 + 30W) AMC + RTM = 457 cm2 (6U VMEbus = 345 cm2) Mechanical key AMC / RTM panels according to rugged uTCA spec (insertion force) Special backplane (zone 1&2) in preparation Backplane in “zone 3” possible Two types of clocks: High precision radial (TCLK[A/B] as defined in AMC.0) Low speed bussed (ports 17-20) Still some issues to be resolved. (e.g. Control of RTM fans) First products available: Elma, Schroff: crates SIS8300: ADC (10 channel, 16 bit, 125 MSPS) with RTM

uTCA backplane for physics Taken from draft document (https://xtca-for-physics.web.cern.ch/xtca-for-physics/MTCA.4.pdf): NOT YET APPROVED A possible configuration: Port 0 should be used for base Ethernet interface Port 1 is connected to the optional (redundant) MCH Port 4 to 7 should be used for PCIe Port 12 to 15 may be used for application specific wire-ring Ports 17 – 20 should be wired as a bus FCLKA (Clk3) should be used for PCIe clock distribution as defined in MTCA.0 R1.0 (main MCH only) TCLKA and TCLKB should be wired from the main MCH to all AMC slots TCLKC and TCLKD should be wired from the redundant MCH to all AMC slots

xTCA for physics ATCA xTCA S/W Mechanical standards: 90% done Zone 3 connector (for RTMs) defined 3 ADF/ZD connectors (40 LVDS pairs, 5 GHz) for data transfer 1 combined connector for: Power: 12V, 50W (+ 3.3V management power) Management (I2C, JTAG, card detection) Mechanical keying Definition of timing signals via Zone 2 connector For dual-dual star topologies xTCA S/W No physics standard planned but recommendations in preparation Mechanical standards: 90% done Specification for clock distribution: 90% done Submission to PICMG for approval planned for September Questionnaire circulated to identify current xTCA projects

xTCA users & products ATCA uTCA / AMC Several plasma (fusion) experiments ITER not yet fully convinced, will also evaluate PXIe XFEL LLRF control system Panda, BELLE II experiments PCIe switch blade developed by IST (Lisbon) uTCA / AMC XFEL LLRF control system (non redundant) DESY did a physics compliant, general purpose AMC with RTM Offer to share design or sub-systems (e.g. MMC)

xTCA projects mentioned @ RT2010 Development of Experiment Link ATCA CPU blade (Based on ATX mother board) COMPASS tokamak (http://portal.ipfn.ist.utl.pt/rt2010/Conference%20Presentations/5-Friday/8h30/3%20-%20CMS-2%20Invited_Horacio_rev1.ppt) ATCA ADC blade IPMC (Intelligent Platform Management Controller) Nuclear fast fusion plant http://portal.ipfn.ist.utl.pt/rt2010/Conference%20Presentations/5-Friday/8h30/4%20-%20CMS2%20RT2010-Exist00_IST%20talk.ppt Timing & Trigger AMC XFEL http://portal.ipfn.ist.utl.pt/rt2010/Conference%20Presentations/4-Thursday/15h00%20-%2015h45%20-%20MO-6/10-PDAQ-42-Hidvegi.pdf ATCA blades XFEL LLRF controls http://portal.ipfn.ist.utl.pt/rt2010/Conference%20Presentations/4-Thursday/15h00%20-%2015h45%20-%20MO-6/16%20-%20PFE-21-D.Makowski_Prototype_Real-time_ATCA-based_LLRF_Control_System.pdf 10 Gbit uplink AMC PANDA (FAIR) http://portal.ipfn.ist.utl.pt/rt2010/Conference%20Presentations/2-Tuesday/15h00-15h45%20-%20MO-4/07%20-%20pdaq-8-slides.ppt ATCA computing blade http://portal.ipfn.ist.utl.pt/rt2010/Conference%20Presentations/2-Tuesday/13h30%20-%20MO-3/15%20-%20PDAQ_40_D.Munchow_RT2010PPT.pdf http://portal.ipfn.ist.utl.pt/rt2010/Conference%20Presentations/1-Monday/16h30%20-%20ATCA%20New%20Standards%20(xTCA)/5%20-%20ATCA-5%20_Z.A%20Liu_%20LisbonTalk.ppt Several ATCA blades Fusion Control http://portal.ipfn.ist.utl.pt/rt2010/Conference%20Presentations/1-Monday/16h30%20-%20ATCA%20New%20Standards%20(xTCA)/2%20-%20ATCA_M.Correira_rt2010_final_MiguelCorreia.ppt http://portal.ipfn.ist.utl.pt/rt2010/ATCA%20Workshop%20Presentations/Lisbon%20xTCA%20Workshop%20Presentation-Program-Abstracts_052710/2-2.1%20XTCA%20WS%20ATCA%20for%20Fusion%20Plasma%20Control%20BGoncalves.pdf AMC carrier http://portal.ipfn.ist.utl.pt/rt2010/Conference%20Presentations/1-Monday/16h30%20-%20ATCA%20New%20Standards%20(xTCA)/4%20-%20ATCA_4_D.Makowski.pdf AMC with FMC site http://portal.ipfn.ist.utl.pt/rt2010/ATCA%20Workshop%20Presentations/Lisbon%20xTCA%20Workshop%20Presentation-Program-Abstracts_052710/2-4.2%20XTCA%20WS%20Versatile%20FPGA%20AMC%20for%20xFEL%20PVetrov.pdf Timing AMC http://portal.ipfn.ist.utl.pt/rt2010/ATCA%20Workshop%20Presentations/Lisbon%20xTCA%20Workshop%20Presentation-Program-Abstracts_052710/2-1.3%20XTCA%20WS%20AMC%20Physics%20Timing%20Distribution%20AHidvegi.pdf IPMI XFEL LLRF http://portal.ipfn.ist.utl.pt/rt2010/ATCA%20Workshop%20Presentations/Lisbon%20xTCA%20Workshop%20Presentation-Program-Abstracts_052710/2-2.3%20IPMI_development_for_DESY_LLRF_System_D.Makowski.pdf ATCA switch Proposed to ATLAS http://portal.ipfn.ist.utl.pt/rt2010/ATCA%20Workshop%20Presentations/Lisbon%20xTCA%20Workshop%20Presentation-Program-Abstracts_052710/2-2.4%20xTCA%20%20WS%20Lessons%20learned%20on%20MPP%20DAQ%20Design_V2%20MHuffer.pdf uTCA system Accelerator controls http://portal.ipfn.ist.utl.pt/rt2010/ATCA%20Workshop%20Presentations/Lisbon%20xTCA%20Workshop%20Presentation-Program-Abstracts_052710/2-4.1%20xTCA%20WS%20Architecrures%20for%20XFEL%20Controls%20%20TJezynski.pdf

IPMI Nobody likes it; everybody uses it PICMG have added features to IPMI for xTCA systems State machine for hot swapping Power budgeting E-Keying Cooling policy (fan control commands) F/W upgrade IPMI spec (600 pages) very ambiguous. This often leads to interoperability problems as manufacturers interpret the spec in different ways DESY did a “dummy” ATCA AMC carrier as a platform for their IPMC development. They have also built a real-time IPMI analyzer

IPMI design Tips Taken from slides of Pete Hawkins (Performance Technologies) Always Provide a Debug Interface Serial Port and/or JTAG Temperature Sensors Remember Power and Ground Planes Conduct Heat PCB mounted sensors don’t measure ambient temperature, but that is how everyone implements them Headroom is More Important Than Absolute Temp Max Temp – (Ambient + Conduction + Heat Dissipation) Thresholds Don’t be Overly Aggressive “Non-Critical” thresholds are intended to just be informative, but people tend to think they mean something is wrong Be Especially Careful with “Non-Recoverable” Thresholds Management software may take action to prevent damage –i.e. may turn off power to a FRU Especially true for temperature sensors

IPMI pitfalls Taken from slides of Pete Hawkins (Performance Technologies) Management Power Budget 15W for ATCA Boards, which includes AMCs for carriers and conversion losses 150mA for AMCs In-Rush Current Applies to Management Power and Payload Power AMC.0 limits must be followed IPMB Capacitive Loading / Hot Swap Glitches Follow the PICMG specs carefully IPMI Controller Resets Payload Function cannot be affected Some I/O needs to be latched Payload Controls, E-Keying Controls, LEDs, etc. Power Boundaries IPMI Controller can be powered while payload is not Connections between payload and IPMI controller may need isolation PCIeReset / FCLKA PCIeRequires PERST# to remain active for 100 μs after the reference clock is stable FCLKA is Part of AMC edge connector, but PERST# is not Follow Requirements in AMC.1 R2.0 carefully 0000h vs. FFFFh IPMI often uses 0000h to mean first item and FFFFh to mean end of list Programmers are used to using null pointers to end lists

Other Information Exhibitors (only ~5) Other conference highlights CAEN still not convinced by xTCA also hesitating to go for VXS VMEbus 2eSST in the pipeline Agilent VXS based 6U high speed ADC (RocketIO on VXS) Schroff / Elma “uTCA for physics shelves are useful not only for physics” Other conference highlights As many new VMEbus designs as xTCA modules Use of GPUs in trigger systems High precision timing systems High speed ADCs

Links & events xTCA Interoperability Workshop xTCA workshop slides Week of October 18, 2010, Schroff Training Center at Straubenhardt, Germany xTCA workshop slides http://portal.ipfn.ist.utl.pt/rt2010/ATCA%20Workshop%20Presentations/Lisbon%20xTCA%20Workshop%20Presentation-Program-Abstracts_052710/ RT2010 presentations http://portal.ipfn.ist.utl.pt/rt2010/

TWEP2010 & xTCA work group meeting a short (personal) summary

xTCA for physics Other observations Standards (PICMG 3.8 & MTCA.4) still in draft status but submission for ballot may happen soon Still a lot of work to be done on the S/W side (such as replacing licensed S/W by open source code) Apparently there are also efforts to standardize ATCA RTMs in Telecoms Wiener also planning to develop a uTCA (MCA.4) shelf Other observations uTCA proposed for XFEL, FAIR and Linac upgrade @ SLAC Lots of good results from signal integrity tests at xTCA frequencies Still lots of VMEbus based designs (also for long term projects) One VXS card (GANDALF for COMPASS) but nobody really compared VXS and uTCA / ATCA

xTCA projects mentioned @ TWEP 2010 Development of Experiment Description AMC with MMC on mezzanine CMS HCAL upgrade (sLHC) MCH Secondary MCH for switching of custom protocol AMC LHCb Read-out board for sLHC MHC mezzanine ATCA blade ATLAS ROD for Pixel/CSC upgrade. New RCE for meshed backplanes (no switch blade) in the pipeline RTM for SIS8300 / DESY AMCs XFEL experiments MTCA.4 RTM Lar ROD (for phase II) “GOLD” card for L1 trigger

xTCA Work Group meeting J. P. Cachemiche MMC soon ready (stable) for use in other projects Modifications for MMC F/W of DESY and CPPM not synchronized CPPM willing to manufacture boards Request for CERN to manage user support R. Larsen New requests for PICMG 3.8 (RTM) functionality slow down completion of the standard and may lead to a proliferation of connector types Still discussions on timing distribution in ATCA Discussions on a standard way of reconfiguring FPGAs in radiation environments (may require fast LAN add-on to IPMI) G. Iles The PCI clock makes the uTCA backplane non symmetrical. Apparently the MTCA.4 team did not spot this Meeting results Participants agreed to found a “xTCA Interest Group” Plan to meet twice per year (during appropriate conference or via EVO) The xtca-news@cern.ch e-mail list will be used for communication A Web site (https://twiki.cern.ch/twiki/bin/view/Sandbox/XTCAInterestGroup) has been set up at CERN. All developers of xTCA H/W & S/W will be invited to add a few words about their projects