1. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) – Networking RCEs RCE Training Workshop Matt Weaver, weaver@slac.stanford.edu 15 June, 2009

2. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 2 Outline ATCA Networks –Zone 2 Base + Fabric –Redundancy traded for multiplicity/bandwidth CIM Hardware Design –Switch ASIC Properties –Switch mapping to Zone 2 and Zone 3 –Rear Transition Module (RTM) –Switch Host Processor CIM Software –Development environment –Protocols CIM Usage –ATCA Hub slot mapping to RCEs –Front Panel Features

3. SLAC Particle Physics & Astrophysics ATCA Networks

4. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 4 Zone 2 Base + Fabric CIM serves as a hub in the dual-star networks CIM Base supports –1G serial ethernet (SGMII) for RCE –10/100/1000 BaseT ethernet for commercial CIM Fabric supports –10G ethernet (4-lanes XAUI) Dual-Star

5. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 5 Redundancy Traded for Multiplicity/Bandwidth Dual Star topology introduced to provide redundancy for maximizing uptime in telecom environments Data acquisition environment can make use of 2x bandwidth from each node Requires extra connectivity between hubs and outside the shelf Dual-Star

6. SLAC Particle Physics & Astrophysics CIM Hardware Design

7. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 7 Switch ASIC Properties The current generation Cluster Interconnect Fabric is based upon the Fulcrum Micro FM2224 ASIC –24-port Ethernet Layer 2 switch –Each port 1Gb / 2.5Gb / 10Gb capable –Fully provisioned, non-blocking, shared memory –Low-latency cut-through operation (200ns) –Pause frame flow control –1.5W per channel –Hardware MAC address lookup, static & dynamic –Supports trapping of link protocol messages Compatible with RCE MGT serial 1 (2.5)Gb

8. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 8 Switch Mapping to Zone 2 and Zone 3 FM2224 Fabric network 1-GE 10-GE XFP Front BoardRTM 10-GE 1-GE 10-GE XFP 10-GE (external network) (external nextork) (external network) FM2224 Base network ( Dual Star Backplane Channels ) ( Neighbor Hub Interconnect ) ( Neighbor Hub Interconnect ) 13 5 4 3 2

9. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 9 Rear Transition Module (RTM) Bulk cabling to external network is in the rear Technology choice is largely independent of front-board design –CX4 electrical connections Inexpensive, Passive short range (~15m) –XFP optical transceivers Pluggable transceivers Short range or long range ($250-$1000) ~1W / channel Total external bandwidth limited by panel real estate –8 XFPs or 8 CX4 –High density optical technology becoming available (20-40)

10. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 10 Switch Host Processor One host processor for managing both network switches –Implemented in Xilinx Virtex-4 Embedded PPC405 (a la RCE) –128MB DRAM –32MB flash memory file system –FM2224 management bus interface –Logical CPU port with DMA Engine on each network –XFP and BaseT PHY management bus masters –IPM Controller for ATCA electronic-keying

11. SLAC Particle Physics & Astrophysics CIM Software

12. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 12 Development Environment GNU C++ cross-platform programming RTEMS 4.9 Same as RCE –Similar hardware –Inherits as much core software and tools as possible Most likely developments will be to implement new management protocols or enhance monitoring

13. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 13 Management Protocols IGMP snooping (multicast control) implemented –Limit forwarding of multicast data to only interested ports –Lessens burden on resource limited NICs IPMI electronic-keying implemented –Interfaces with ATCA Shelf Manager to achieve safe integration with commercial board Fabric channels Large network management protocols not yet implemented –Desirable for large sets of switches { RSTP, LACP } –Not difficult, just hasn’t been a development priority

14. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 14 User Configuration DHCP Vendor Encapsulated Options –IP address assignment of both network host ports –Switch channel configurations (in absence of Shelf Mgmt) –VLAN port assignments –Static MAC address assignments Terminal -> Flash memory –Manual changes applied via the shell –Accumulated configuration is written to flash memory

15. SLAC Particle Physics & Astrophysics CIM Usage

16. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 16 ATCA Hub Slot Mapping to RCEs

17. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 17 Front Panel Features – Rotary Switch and Reset Button 16-position rotary switch –Configuration selection –Read following reset Hard reset button

18. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 18 Front Panel Features – Auxiliary Ports XFP and RJ45 external connectivity to each network Convenience for test-stand setups Fabric Network Base Network RJ45 XFP

19. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 19 Rear Panel Ports High throughput external connectivity 5 x10Gb (Fabric), 3 x10Gb (Base) Fabric P3-7 Base P3-5 Example XFP RTM

20. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 20 Front Panel Features – Multi Function Display CIM – following reset DHCP – attempting DHCP B … - Base channels status F … - Fabric channels status enabled link up enabled link up Fabric or Base Zone 2 Front Panel Zone 3 Slot 1/2 Slot 3Slot 4Slot 5 Slot 14 P1P2 P3P5P6P7P4 H-AH-CH-B

21. SLAC Particle Physics & Astrophysics The Cluster Interconnect Module (CIM) 21 Summary CIM is designed to make full use of ATCA provided bandwidth –Dual 10Gb and dual 1Gb channels to each slot –Several external 10Gb channels (6/4) –Fully provisioned packet switching fabric CIM Software –Same development environment as RCE –Customizable for large scale aggregation