Status of Compute Node Zhen’an Liu, Dehui Sun, Jingzhou Zhao, Qiang Wang, Hao Xu Triglab, IHEP, Beijing Wolfgang Kühn, Sören Lange, Univ. Giessen Belle2 TDAQ 2011, Beijing, Jan , 2011

H.Xu 26/1/20112 Outline  Compute Node for PXD Upgrade from present version 4 boards mini-production AMC-based new design  Example firmware development  Summary

H.Xu 26/1/ Compute Node for PXD IGAS- IHEP Giessen ATCA System  Requirements of PXD DAQ 6Gbps lane rate per link 2x3Gbps/1x6Gbps 4GB DDR2 Memory per FPGA  Approaches Modification from present version 3.125/6.5Gbps compatible New design AMC/xTCA based new design  Mini-Production 2 boards assembled with virtex4 fx60-11 for SFP+ -> For PXD DAQ 2 boards assembled with virtex4 fx60-10 for SFP -> For PANDA DAQ

H.Xu 26/1/20114 Test of New Board  Data flow Optical link->DDR2 SDRAM -> PC  Optical Tested about 1 hour, estimated BER< 1.0x10^-14  P2P transmission on backplane Failed to transmit limited by bandwidth of backplane Eye-diagram of optical link Eye-diagram of backplane transmission

H.Xu 26/1/20115 AMC-based New Design  Motivation  Introduction to AMC  Development of Carrier Board and AMC Modules

H.Xu 26/1/20116 Motivation  Compute Node A general high availability and high performance data acquisition and trigger system  But, not satisfied in Memory capacity Timing and control capabilities xTCA for physics compliant (IHEP is a co-sponsor of this standard)  xTCA is a good choice for next generation 1 ATCA Carrier Board with high bandwidth switch + AMC modules Custom AMC modules for different applications Flexibility to reuse and upgrade Flexibility for maintaining

H.Xu 26/1/20117 What is AMC?  Advanced Mezzanine Cards are printed circuit boards (PCBs) that follow AMC.0 R2.0 specification of the PCI Industrial Computers Manufacturers Group (PICMG). AdvancedMC defines a modular add-on or “child” card that extends the functionality of a Carrier Board. AdvancedMC Modules lie parallel to and are integrated onto the Carrier Board by plugging into an AdvancedMC Connector.  Envisioned AdvancedMC Modules cover a wide range in terms of their functionality and include the following examples: Telecom connectivity Processors (CPUs, DSPs, and FPGAs) Network communication processors Network communications co-processors Mass storage From PICMG

H.Xu 26/1/20118  PICMG 3.0 optimized: All elements must work within the bounds of the PICMG 3.0 base specification and build upon its strengths of Reliability, Availability, and Serviceability (RAS).  Building block for MicroTCA: After the initial release of this specification, PICMG developed the MicroTCA specification, PICMG MicroTCA.0, which uses AMC Modules plugged directly into a backplane.  System management: System management is an extension of the PICMG 3.0 Shelf management scheme.  Hot Swap support: Hot Swap of AdvancedMC Modules is enabled in support of Availability and Serviceability objectives.  LVDS interconnect: AdvancedMC is optimized for LVDS interconnects.  Low pin count: The interconnect is conservative in its total pin count, thereby reducing the amount of space required on both the Module and the Carrier Board, yet provide sufficient real estate for intended interconnects and usage models.  Reduced development time and costs: The reduced total cost of ownership is accomplished through component standardization and by driving economies of scale.  Modularity, flexibility, and configurability: AMC Modules have designed-in modularity features with the physical sizes that offer flexibility in use and configuration on an AMC Carrier AdvancedTCA Board including the ability to stack mezzanines.  Future advances in signal throughput: AMC technology anticipates advances in interconnect technologies by supporting a minimum of 12.5 Gbps throughput per LVDS signal pair. Characters of AMC

H.Xu 26/1/20119 Module Sizes and Width  six types of Module available A Full-size Module is the most common, allowing up to mm high components A Mid-size Module allows component heights maxed at to mm A Compact Module allows only 8.18 mm. A special carrier card known as hybrid or cutaway carrier is required to hold one Full-size Module or two Compact- size Each height is paired with a width, single (74mmx180mm) or double (149mmx180mm), describing how many carrier slots the board fills. A double width card allows more component space, but does not provide any additional power or bandwidth because it only uses a single connector. From PICMG AMC.0 R2.0

H.Xu 26/1/ AdvancedMC Connector  The various connector mount types are available for all AMC Connector styles, B, B+, AB, and A+B+.  Fabric Interface 40 signal pairs allocated to the Fabric Interface  System Management Interface 9 contacts allocated to the System Management Interface  AMC Clock Interface 5 signal pairs allocated to the AMC Clock Interface  JTAG Test Interface 5 contacts allocated to the JTAG Test Interface  Power/ Ground 8 contacts allocated to Payload Power 56 contacts to allocated to Logic Ground From PICMG AMC.0 R2.0

H.Xu 26/1/ Hardware Platform Management  Management is performed through IPMI messaging over an onboard IPMB referenced throughout this specification as IPMB-L. Each Module has a unique IPMB-L address derived from its Geographic Address.  Carrier provides ways to isolate the IPMB-L connection to each Module.  MMC: Module Management Controller From PICMG AMC.0 R2.0

H.Xu 26/1/ Power Distribution  Payload power (AMC connector) +12V(10.8V to 13.2V) Max. 80W,  Management power +3.3V +/- 0.3V Max. 150mA, At least 165mA(Carrier) BUT  ATCA power supply: 200W/slot x80% = 160W Only ~35W for each module if there are 4 AMC modules ~55W for each module by increasing power to 300W/slot ( PIM300 + power module) Cooling challenges for 300W+ power supply From PICMG AMC.0 R2.0

H.Xu 26/1/ Development of Carrier Board  Carrier Board with high bandwidth switch for neighbor-link Virtex-4 FX60 based – functions test 1 Virtex-4 FX60 FPGA 2GB DDR2 SODIMM 400Mbps 512Mb FLASH Memory 13x to backplane 2x Gbit Ethernet PCB layout is ongoing Virtex-6 based – high performance 1 Virtex-6 FPGA 2/4GB DDR3 SODIMM 800Mbps 512Mb FLASH Memory 13/26 to backplane 2xGbit Ethernet

H.Xu 26/1/ Carrier Board Rev.1 Clock/Timing from backplane are signals of XTCA for physics

H.Xu 26/1/ AMC Module 1 Please see Jingzhou’s talk: AMC processor development

H.Xu 26/1/ AMC Module 2  Spartan6-based –Lowest cost, lowest power

H.Xu 26/1/ Example Firmware Development  Open source Linux  Linux boot and reconfiguration online  Data flow Data transmitted with LocalLink are under the control of PLB master peripheral and DMA controller. Status information are connected to PPC through PLB.

H.Xu 26/1/ Function Test System NFS Event Gen..cxx Data token.C macro Detector data.hex Reconstruction.cxx Data analysis.C macro framework Software COPPER PMC CPU FINESSE ComputeNode optical

H.Xu 26/1/ Summary  Test results of new CN for PXD show that the boards work well  Optical is stable  AMC-based new design is on the way  Example firmware for data transmission and readout developed

Thanks for you attention !

Backup Slides