1. RCE Platform Technology (RPT)v4
RCE Platform Technology (RPT)
ATCA and the COB
Gregg Thayer

2. Outline ATCA COB Advanced Telecommunications Computing Architecture
PICMG 3.0 Standard
Developed by the telecom industry
Features
High Availability
Redundancy
Hot Swap
Uses Integrated Platform Management (IPM)
Rear Transition Module
Separates physical data interface from processing
High-speed, protocol agnostic backplane
COB
Cluster On Board
Designed and built at SLAC
Compliant ATCA Front Board

3. The ATCA Shelf The ATCA Shelf Provides Monitoring and Management
Superset of Integrated Platform Management (IPM)
Power and Cooling
Up to 400W/slot
AC and DC options allow rack power aggregation
Power options are vendor specific
Fans provide cooling
Intra-shelf data transport
Base Interface
Fabric Interface
Synchronization Clock Interface
Update Interface
2 to 16 Slot configurations
Small shelves usually have horizontally oriented slots
Large shelves often have vertically oriented slots
14 slots is the largest to fit in a 19” rack
16 slot Euro standard uncommon
High reliability supported by redundant systems
Slot addressing (1 – 16)
Physical: Left to Right or Bottom to Top
Logical: Vendor specific
Use Physical Address when referring to slots

4. ATCA Slot Front Board Rear Transition Module (RTM)
Management Controller (IPMC)
Payload
Power
Management Power
3.3V always on
Powers IPMC and all other management functions
Payload Power
Negotiated with Shelf Manager
Rear Transition Module (RTM)
Powered by Front Board
Has no separate IPMC

5. ATCA Backplane The ATCA Backplane is divided into 3 Zones
Zone 1 – Power and Management
Power from dual-redundant -48VDC supply rails
System Management via redundant IPM Bus (IPMB) connection to Shelf Manager
Hardware Address
Zone 2 – Data Transport Interface
Protocol agnostic
Up to 200 Differential pairs
Up to 10Gbps signaling
Backplane inter-slot topology varies by interface
Zone 3 – Rear I/O
Connects Front Board to RTM
Not defined by standard
RTM must be powered through Front Board

6. Shelf Management The Shelf Manager watches over managed devices
Retrieves inventory information, sensor readings
Receives event reports and failure notifications from Boards
Reports anomalous conditions
Takes whatever corrective actions it can
Altering cooling fan speed
Deactivating boards
Managed devices are Field Replaceable Units (FRUs)
ATCA Front Boards and RTMs are FRUs
FRUs communicate with the Shelf Manager through their IPMC
Called an Intelligent FRU
FRUs may be represented by the IPMC of another FRU
Called a Managed FRU
RTM is a Managed FRU represented by the IPMC on the Front Board
Communication between the Shelf Manager and the IPMCs is done over the IPM Bus (IPMB)
Two-way, redundant, I2C distributed over the ATCA backplane (Zone 1)
Topology can be bussed or radial
Shelf Manager can integrate into a larger IPM system in a number of ways
Simple Network Management Protocol (SNMP)
Remote Management Control Protocol (RMCP)

7. FRU States
Each IPMC in the Shelf tracks the state of the FRUs it controls
Each FRU can be hot swapped
FRUs spend most of their time in states M1 and M4
Some state transitions are initiated by changes detected by the FRU itself
Insertion and Ejection Criteria
Some state transitions are initiated by commands from the Shelf Manager
Some can be initiated by either
The state of the FRU is not the state of the Payload
Payload operation largely takes place while the FRU is in state M4

8. Shelf FRU Information
Every Shelf stores information describing itself and its capabilities
Storage location is vendor specific, but is often non-volatile storage located on the shelf itself
Readable by and through the Shelf Manager
The Shelf FRU Information contains (among other things)
Shelf Address (name)
Address mapping table
Physical Address
Logical Address
IPMB Address
Hardware Address
Data Transport Backplane topology
Shelf Manager Network configuration (optional)
Power and cooling capabilities of the shelf
Can be extended with custom records

9. ATCA Hot Swap, Power Negotiation and E-Keying
Boards can be inserted and removed from the Shelf while the system is live
The Shelf Manager is responsible for negotiating with IPMCs when FRUs enter and leave the system
Power Negotiation
The IPMC is powered by a 3.3V always present Management Power
Separate from payload power
When a FRU is inserted, the Shelf Manager inquires as to its power needs
A FRU can be capable of multiple power levels
The Shelf Manager retrieves the power capabilities of the shelf from the Shelf FRU Information
If the needs of the Front Board can be supported, the Shelf Manager allows the IPMC controlling the FRU to activate the payload power
Electronic Keying
The Shelf Manager requests the Data Transport capabilities of each Front Board
The Shelf Manager retrieves the backplane mapping from the Shelf FRU Information
Combining these, the Shelf Manager sends commands to enable all compatible channels
When a board is inserted or extracted, E-Keying will cause the commensurate configuration changes in the boards with which it communicates over the Data Transport Interfaces

10. Data Transport Backplane Topologies
Dual Star
Each of two Hub slots connect to every other slot
Implies two types of boards: Hub and Node
Hub boards must occupy Logical Slots 1 and 2
Full Mesh
All slots connect to all other slots
Mesh capable boards can be used as either Hub or Node boards in other topologies
Can be used to implement all other non-replicated topologies
Replicated Mesh
Multiple connections between slots
Pairs of slots need not share same number of replications
Increases capacity, doesn’t change connectivity
Other Topologies
Dual-Dual Star
Multi-plane Switch

11. Data Transport Interfaces
The specification assumes boards need to interact with one another
Defines four different types of board communication
The Base Interface
The Fabric Interface
The Synchronization Clock Interface
The Update Interface
The specification is as protocol agnostic as possible
Only determines connectivity between boards
Assumes data is transmitted/received serial-differential
Assumes communication is full-duplex (independent transmit and receive)
Limitations to agnosticism
Board must provide IP support on either Base or Fabric Interface
The Base Interface (if implemented) must satisfy an Ethernet MAC
10/100/1000BaseT

12. Data Transport Interface Channels
All point-to-point transport interfaces are characterized in terms of channels
A Channel is a group of differential signal pairs that are physically routed together on the Backplane to provide an interconnect trunk between two Slots
The number pairs per channel and maximum number of channels varies by interface type
A Base Channel consists of 4 differential pairs
A Fabric Channel consists of 8 differential pairs
An Update Channel consists of 10 differential pairs
Slot interconnect topology varies by interface type
The Synchronization Clock Interface is bussed and uses 6 differential pairs

13. Data Transport – Fabric Interface
The Fabric Interface is comprised of 15 Channels providing connectivity among up to 16 boards in a Full Mesh or Star configuration
120 differential pairs
Front boards must be capacitively coupled to the backplane to isolate transmitter and receiver common mode voltages
Limits protocols to DC balanced signals
The IPMC should be able to disable transmitters as part of the E-Keying process
When disabled, transmitters do not transmit signaling voltages to the backplane
The Fabric Interface can be partitioned into multiple fabrics among boards
The Dual Star can be used to support two distinct, redundant fabrics by placing hub boards in Logical Slots 1 and 2
Replicated Mesh works similarly in shelves with fewer than 9 slots
Replicated Mesh configurations can also be used to increase capacity in a single fabric
All of this is obviously contingent on the capabilities of the Front Boards in the system
ATCA ensures that only compatible channels are enabled through the Electronic Keying (E-Keying) process

14. Data Transport – Base Interface
The Base Interface is comprised of 16 channels providing 10/100/100BaseT Ethernet connectivity among 16 boards in a Dual Star configuration and an optional connection to the Shelf Manager
64 differential pairs
The Base Interface drivers do not need to be isolated from the backplane
The Ethernet PHYs are allowed to auto-negotiate prior to system management enable (E-Keying)
Base Interface is still subject to E-Keying negotiation

15. Data Transport – Synchronization Clock Interface
The Synchronization Clock Interface provides a set of clock busses to enable applications that require the exchange of synchronization timing information among multiple boards in a shelf
Each bus is a multi-drop, differential pair
For redundancy, six busses are divided into three redundant groups
CLK1 (A&B) – Telecom Specific
8 kHz A/B failover (Digital Telephony)
CLK2 (A&B) – Telecom Specific
19.44 MHz A/B failover (SONET reference clock)
CLK3 (A&B) – User defined
A and B can be used independently, but limited to 100MHz
Usage is problematic in multi-tenant systems
E-Keying is key to self-consistent configuration
Negotiates which boards drive the bus and which listen
Resolves conflicts among multiple master requests

16. The Cluster On Board

17. Cluster on Board (COB) Data Transport
ATCA Front Board
Base Interface Node Board
Fabric Interface Mesh Board
10G Ethernet
Synchronization Clock Interface
Payload Function
Hosts a Cluster of RCEs
On mezzanine boards
Decouples COB development from mezzanine development
Cluster Interconnect
Connects all RCEs
Faceplate SFP+
Connects to other clusters over ATCA Fabric Interface
Timing Sources
ATCA Synchronization Clock Interface
External through Rear Transition Module
Internally Generated COB

18. RCE Synchronization The DTM can distribute timing signals
To DPMs through fan-out on COB
To RTM for external transport
To the ATCA Synchronization Interface for intra-shelf timing
The timing signals can originate
Internal to the DTM
For simulating external timing
For local COB synchronization
On the RTM
On RTM Mezzanine Board (RMB)
From an external source
From the ATCA Synchronization Interface

19. Cluster on Board (COB) Management
ATCA Front Board
Management via IPMC
Power Negotiation
E-Keying
Temperature Sensing
Payload Function
IPMC controls the payload over I2C busses to functional components located in Bays
COB IPMC is based on software licensed from Pigeon Point Systems
Extended and modified to support COB payload functions
IPMC communicates to RCEs through the Bootstrap Interface (BSI)
IPMC controls the RCE with General Purpose I/O (GPIO) I2C devices on the DPM/DTM
Status Lines
Reset Line
Power Usage
Extensive monitoring of temperatures, voltages, and currents

20. RCE Bootstrap Interface (BSI)
IPMC uses the BSI to coordinate the RCEs in a Cluster
The RCE contains an I2C slave that is connected to the IPMC
Visible to the IPMC as a 2 kByte register space
The RCE signals the readiness of this interface by asserting a signal connected to the GPIO device on the DPM/DTM
The interface is not ready until the RCE provided information below has been written
Information provided by RCE
BSI Version
Network PHY type
CE MAC Address
CE Interconnect Definition
RCE Status
Information provided to RCE
Mezzanine board serial number (from ID PROM)
Cluster Address (Slot/Bay/RCE)
Cluster Group Name (Shelf Address)
External Interconnect Definition (From RTM)
Information provided to RCE on DTM
Cluster Switch Configurations
From network PHY types for intra-COB links
Results of E-Keying for inter-COB links
Presence of FP SFP+ transceivers
CE Interconnect Definitions
Shelf IP Information

21. Rear Transition Module (RTM)
Connects to COB Zone 3
Power and Management on blue connector
Most management function contained on a COB standard daughter card
Physical adaptation layer for DPM signals
96 CML, full duplex lanes driven by MGTs on DPM RCEs
External interface may be application specific
RTM Mezzanine Board (RMB) connects to DTM
Doesn’t actually need to be a mezzanine
6 Pairs of LVDS signals to DTM
4 pairs to clock capable I/O
2 pairs to general purpose I/O
2 pairs of CML signals to DTM MGT (Tx/Rx)

22. Do-it-yourself RTM
RTMs are the adaptation layer between the front-end and the COB
It is expected that some users will need to design their own RTM
SLAC will have built a few “generic” RTMs which may also serve
There are only a few requirements to build a COB compatible RTM
Obey the COB Zone 3 pinout
Include the Management (I2C) Daughterboard
This is supplied by SLAC
Provides all COB required Management functionality
Must include ATCA specified Face Plate devices
Hot Swap Handles
LEDs
SLAC intends to create an RTM kit which will include what is needed to build an RTM
Mechanical drawings
I2C Management daughter board
COB Zone 3 pinout description
Bill of Materials

23. COB and RTM

24. DPM0 DPM1 DPM3 DPM2 RTM CI DTM COB and RTM Zone3 Zone2 IPMC
SFP+
IPMC
Power Conversion
Zone1

25. COB and RTM BAY0 BAY1 BAY5 BAY3 BAY2 BAY6 BAY4 RCE0 RCE0 RCE2 RCE2

26. Data Transport Module (DTM)
1 RCE
SD card stores all code to operate RCE
SOC configuration file
RCE Core Software
Application specific software
Manages the Cluster Interconnect Switch
Connected to ATCA Clock Synchronization Interface
Connected to RMB
Connected to ATCA Base Interface
Connected to DPM timing fanout
Connected to DPM consoles and JTAG

27. Data Processing Module (DPM)
2 RCEs
SD card stores all code to operate RCE
SOC configuration file
RCE Core Software
Application specific software
12 lanes of MGT per RCE to RTM
4 lanes of MGT per RCE to CI
Timing interface per RCE
1 pair from DTM to MGT reference clock
2 pairs from the DTM to clock capable I/O
1 pair feedback from user I/O to DTM
Serial Console and JTAG
Connected to DTM

28. IPM Controller (IPMC) ATCA Functions Cluster Configuration
Communicates with Shelf Manager
Power Negotiation
Hot-Swap
E-keying
Temperature Control
Cluster Configuration
Controls power and reset lines of RCEs
Communicates cluster configuration information to RCEs

29. COB Activation When a COB is inserted into a Shelf
The Management Power is applied and the IPMC boots
The IPMC requests Shelf FRU information from the Shelf Manager
Uses Shelf Address Map to determine the Physical Slot number
Retrieves Shelf Address (shelf name)
Retrieves Zone 2 backplane topology to forward to DTM
Retrieves Cluster IP information
The IPMC requests RCE provided information from the BSI of each RCE
The IPMC requests FRU Information from the RTM
Type, Power requirements
When the handle switch is closed the IPMC requests permission to activate from the Shelf Manager
Shelf Manager and IPMC negotiate Power
When Payload power is applied
The COB FRU enters the Active state
The RCEs configure and boot

30. RCE States
The State of the RCE is not the same as the state of the FRU
The RCE state is constructed from the following bits that the IPMC can read
Mezzanine present (from COB)
Mezzanine payload power enabled (from COB)
Mezzanine reports all voltage regulators OK (from mezzanine GPIO)
SOC Reset line assertion (from mezzanine GPIO)
RCE Ready line asserted (from mezzanine GPIO)
RCE Boot Status (from RCE BSI)
The RCE States are
Not Present
IPMC can detect the presence of COB Mezzanine
Each COB Mezzanine reports the number of RCEs
Powered Off
Prior to payload power application
Voltage Not OK
Each COB Mezzanine monitors the state of its voltage regulators
In Reset
The SOC Reset line is held until the Voltage is OK
Not Ready
Once SOC Reset has been released, the SOC configures and boots
When the BSI is present, the SOC asserts Ready, the IPMC begins reading/loading the BSI with the values required to complete RCE booting
Not Booted
While the RCE boots, it reports a status value to the IPMC
Running
Once Booted, the IPMC continues to monitor the state of each RCE

31. Summary ATCA is a standard that provides solutions for
Monitoring and Management
Power and Cooling
Intra-shelf data transport
The COB is an fully compliant ATCA Front Board
IPMC based on a licensed commercial product
Extended to support our payload needs
Hosts a Cluster of RCEs
Hosts a Cluster Interconnect
Supports synchronous timing
The RTM is customizable to the physical interface of your front-end
A kit will be available to ensure compatibility with the COB