1. AMC-based Upgrade of Compute Node Hao XU xuhao@ihep.ac.cn Trigger group of IHEP, Beijing PANDA DAQT and FEE Workshop, Rauischholzhausen Castle April 2010 II. Physics Institute Justus-Liebig-University Giessen EPC, IHEP, CAS Beijing

2. 14/4/2010H.Xu2 Agenda  Motivation  Introduction to AMC  Development of Carrier Board and AMC Modules

3. 14/4/2010H.Xu3 Motivation  Compute Node An universal high performance platform prepared for multiple applications High availability: ATCA-based (full mesh topology in backplane)‏ FPGA (now Virtex-4)‏, 5 Virtex4 FX60 FPGA, 10 GB DDR2 SDRAM (2GB per FPGA), High Bandwidth, 320MB FLASH (64MB per FPGA)  But, new requirements from more and more applications Larger memory Faster optical link Larger bandwidth for neighbor link xTCA for physics compliant …  It is not good to design one board for all applications Complexity High cost (prototype)  AMC (Advanced Mezzanine Card) is a good choice 1 ATCA Carrier Board with high bandwidth switch + AMC modules Custom AMC modules for different applications flexibility to reuse and upgrade

4. 14/4/2010H.Xu4 What is AMC?  Advanced Mezzanine Cards are printed circuit boards (PCBs) that follow AMC.0 R2.0 specification of the PCI Industrial Computers Manufacturers Group (PICMG). AdvancedMC defines a modular add-on or “child” card that extends the functionality of a Carrier Board. AdvancedMC Modules lie parallel to and are integrated onto the Carrier Board by plugging into an AdvancedMC Connector.  Envisioned AdvancedMC Modules cover a wide range in terms of their functionality and include the following examples: Telecom connectivity (ATM/POS [OC-3/12/48], T1/E1, VoIP, GbE, etc.) Processors (CPUs, DSPs, and FPGAs) Network communication processors (NPUs) Network communications co-processors (Classification, Security or Intrusion Detection) Mass storage From PICMG

5. 14/4/2010H.Xu5 Characters of AMC  PICMG 3.0 optimized: All elements must work within the bounds of the PICMG 3.0 base specification and build upon its strengths of Reliability, Availability, and Serviceability (RAS). The AdvancedMC Module is not limited by other chassis standards.  Building block for MicroTCA: After the initial release of this specification, PICMG developed the MicroTCA specification, PICMG MicroTCA.0, which uses AMC Modules plugged directly into a backplane.  System management: System management is an extension of the PICMG 3.0 Shelf management scheme.  Hot Swap support: Hot Swap of AdvancedMC Modules is enabled in support of Availability and Serviceability objectives. The focus is on front loadable Hot Swap Modules with non Hot Swap being an optional implementation.  LVDS interconnect: AdvancedMC is optimized for LVDS interconnects.  Low pin count: The interconnect is conservative in its total pin count, thereby reducing the amount of space required on both the Module and the Carrier Board, yet provide sufficient real estate for intended interconnects and usage models.  Reduced development time and costs: The reduced total cost of ownership is accomplished through component standardization and by driving economies of scale.  Modularity, flexibility, and configurability: AMC Modules have designed-in modularity features with the physical sizes that offer flexibility in use and configuration on an AMC Carrier AdvancedTCA Board including the ability to stack mezzanines.  Future advances in signal throughput: AMC technology anticipates advances in interconnect technologies by supporting a minimum of 12.5 Gbps throughput per LVDS signal pair.

6. 14/4/2010H.Xu6 Module Sizes and Width  six types of Module available A Full-size Module is the most common, allowing up to 23.25 mm high components A Mid-size Module allows component heights maxed at 11.65 to 14.01 mm A Compact Module allows only 8.18 mm. A special carrier card known as hybrid or cutaway carrier is required to hold one Full- size Module or two Compact-size (see connectors below). Each height is paired with a width, single (74mmx180mm) or double (149mmx180mm), describing how many carrier slots the board fills. A double width card allows more component space, but does not provide any additional power or bandwidth because it only uses a single connector. From PICMG AMC.0 R2.0

7. 14/4/2010H.Xu7 AdvancedMC Connector  The various connector mount types are available for all AMC Connector styles, B, B+, AB, and A+B+.  Fabric Interface 40 signal pairs allocated to the Fabric Interface  System Management Interface 9 contacts allocated to the System Management Interface  AMC Clock Interface 5 signal pairs allocated to the AMC Clock Interface  JTAG Test Interface 5 contacts allocated to the JTAG Test Interface  Power/ Ground 8 contacts allocated to Payload Power 56 contacts to allocated to Logic Ground  2 contacts reserved From PICMG AMC.0 R2.0

8. 14/4/2010H.Xu8 Hardware Platform Management  Management is performed through IPMI messaging over an onboard IPMB referenced throughout this specification as IPMB-L. Each Module has a unique IPMB-L address derived from its Geographic Address.  Carrier provides ways to isolate the IPMB-L connection to each Module.  MMC: Module Management Controller From PICMG AMC.0 R2.0

9. 14/4/2010H.Xu9 Power Distribution  Payload power (AMC connector) +12V(10.8V to 13.2V) Max. 80W, 7.4A(@10.8V)  Management power +3.3V +/- 0.3V Max. 150mA, At least 165mA(Carrier) BUT  ATCA power supply: 200W/slot x80% = 160W Only ~35W for each module if there are 4 AMC modules ~55W for each module by increasing power to 300W/slot ( PIM300 + power module) Cooling challenges for 300W+ power supply From PICMG AMC.0 R2.0

10. 14/4/2010H.Xu10 Development of Carrier Board  Carrier Board with high bandwidth switch for neighbor-link Virtex-4 FX60 based – functions test 1 Virtex-4 FX60 FPGA 2GB DDR2 SODIMM 400Mbps 512Mb FLASH Memory 13x RocketIOs @2Gbps to backplane 2x Gbit Ethernet Schematic is ongoing, PCB will be delivered in June/July Virtex-6 based – high performance 1 Virtex-6 FPGA 2/4GB DDR3 SODIMM 800Mbps 512Mb FLASH Memory 13/26 RocketIOs @6.25Gbps to backplane 2xGbit Ethernet

11. 14/4/2010H.Xu11 Carrier Board Rev.1

12. 14/4/2010H.Xu12 AMC Module 1  Spartan6-based –Lowest cost, lowest power

13. 14/4/2010H.Xu13 AMC Module 2  Virtex6-based – Lowest power, high performance

14. 14/4/2010H.Xu14 AMC Module 3  Dual Virtex6-based – Ultra high performance

15. Thanks for your attention