17 th Real-Time Conference | Lisboa, May 2010 An overview of the ATCA ® timing and synchronization resources for control and data acquisition J. Sousa 1, A.M. Fernandes, A.J.N. Batista, M. Correia, H. Fernandes, B.B. Carvalho, B. Gonçalves 1, C.A.F. Varandas 1 Member PICMG for Physics Committees 1 Instituto de Plasmas e Fusão Nuclear Instituto Superior Técnico, Lisbon, Portugal

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May PICMG consortium is currently working on a extension of the ATCA family targeting Physics experiments I&C (xTCA) New hardware Specifications are being developed to: –Define additional Rear Transition Module connectivity (ARTM and uRTM) –Add additional Timing/Synchronization resources –Better the compatibility between µTCA and ATCA platforms The scope of this presentation is the timing extensions for the ATCA 3.x specification. The following slides will present: –An overview of the available Timing resources on ATCA –The current proposal for Timing on xTCA –Example of implementation on Hub and Node cards (AMC carrier) –Candidate timing network protocols for fabric/external connections –Signal integrity concerns –Test results for a synchronous timing network implementation Introduction

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Timing overPortsLines availablePerformanceCompatibility Bussed Clock Interface CLK1, CLK2 assigned Only CLK3 may be used 6 diff pairsFair 100 MHz bussed Compatible Base interfaceTiming+Data hub in slots 1;2 All nodes receive/send 2 distinct timing signal Good ~1 GHz? P2P Changes to ATCA 3.0 required Shared Fabric port Timing+Data lines share a fabric port All nodes receive/send up to 6 distinct timing signals Good ~3 GHz P2P <100 ps jitter Changes to ATCA 3.x required Fabric star on hub slots Timing hub in slot 3;4 Ports 3;4 on node cards All nodes receive/send up to 8 distinct timing signals Good * ~3 GHz P2P <100 ps jitter Addition of ATCA 3.6 Dedicated backplane lines Update channel ports redefined for timing Up to 10 additional, equal length, low crosstalk diff pairs Excellent # Low skew Low jitter Not compatible if when update channels in use ATCA Timing Resources # When using equal lenght lines on backplanes and zero delay buffers * Can be further improved with the implementation of equal lenght,low crosstalk lines on new backplanes.

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Goal: use current backplanes – compatibility with PICMG 3.x. Proposal: hub on slot 3 (+ 4) with new Link Type for timing signals. Hub can be used on dual-dual star and full-mesh backplanes. The timing hub will provide point-to-point and bidirectional links to each node card and implement the following functions: –local generation or distribution of timing signals to the node cards –route any timing signal from/to any node card (broad/multicast or P2P) The following types of timing signals are to be supported: Timing signals over fabric NameSignal typeDescription BSTBinary State Timingclock, trigger and gate type signals mBSTMultiplexed BSTsynchronous time multiplexed BST signals on a ‘n’- bit word (e.g. 8b/10b -> n = 8) EETEncoded Event and Timing Deterministic timing messages (sequence of words) modulated on a carrier clock. Carry trigger/event signals transport, clock skew correction, and time stamp

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Dual dual-star backplane routing Logical Slot # Con.Ch # P P P P P P P P P P P P P P P SLOT-CHANNEL Row #Interface Designation J22/P22 Connector Pairs (Node board) abcdefgh 9Fabric Channel 3 BST_Tx2[3]BST_Rx2[3]BST_Tx3[3]BST_Rx3[3] 10 EET_Tx[3]EET_Rx[3]BST_Tx1[3]BST_Rx1[3] X4 channel 8 diff pairs

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May ValueDefinition 00h Reserved 01h PICMG 3.0 Base Interface 10/100/1000 BASE-T 02h PICMG 3.1 Ethernet Fabric Interface 03h PICMG 3.2 InfiniBand Fabric Interface 04h PICMG 3.3 StarFabric Fabric Interface 05h PICMG 3.4 PCI Express Fabric Interface 06h PICMG 3.5 Serial RapidlO Fabric Interface 07h PICMG 3.7 Timing over Fabric Interface 08h..EFh Reserved F0h..FEh OEM FFh Reserved e-Keying Type ExtensionDescription 0000b BST: High-Speed LVDS signalling (< 1 Gbit) (ANSI TIA/EIA-644 Standard 0001b BST: CML signalling (< 6.5 Gbit, 4.25 Gbit). 0010b-1011b Reserved 1100b-1111b OEM OptionSignalsPort 0 Port 1 Port 2 Port 3 1 EET X 2 mBST X 3 BST1,2,3 XXX 4 EET + BST1,2,3 XXXX 5 mBST + BST1,2,3 XXXX Port Configuration options up to 3 BST and 1 EET ports: Link TypeLink Extension mBST and EET signaling: adopt PICMG 3.5 (SRIO) PHY (1.25, 2.5, 3.125, 5 Gbaud)

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Timing on AMC Carrier example

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Example of Timing Hub

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Criteria: time jitter and channel to channel skew performance Options: Proprietary (e.g. using FPGA) –Can be tailored for maximum synchronization performance –Higher level synchronization protocols (PTP) and link delay compensation (down to ps) can be straightforwardly added –Example: DESY system wide synchronization (μTCA) for XFEL. IEEE-1588 (PTP) over Ethernet –Synchronization independent of link delay to a degree of accuracy White Rabbit –PTP over Synchronous Ethernet (endpoints recover the COMM clock) –Includes line delay compensation PTP timing over PCIe or SerialRapidIO? –Lower message time jitter –Inherently synchronous, link delay compensation can be added EET candidate protocols

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May “This standard defines a protocol enabling precise synchronization of clocks in measurement and control systems. Networks shall support multicast messaging including but not limited to Ethernet”. Actual synchronization accuracies as low as 50 ns. Not ‘synchronous’. No fine delay adjustment. IEEE-1588 (PTP)

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Uses hardware-assisted PTP for clock synchronization. –Synchronizes local clock with the master clock by measuring and compensating the coarse delay (using PTP) introduced by the link in multiples of 8 ns. –Fine delay adjustment is performed using a digital ‘Dual Mixer Time Difference’ which outputs a measure of the clock loop phase driving an adjustable delay element. All network nodes use the same physical layer clock (125 MHz), generated by the System Timing Master. Clock is encoded in the Ethernet carrier and recovered by the PLL in the PHY. Uses packet preemption and deterministic protocol for timing correction messages. –When timing messages arrives at a switch, the low-priority packet currently being routed is terminated so the HP packet can be sent out with minimal latency. Preliminary AMC protoype measurements: ± 350 ps, STD=80ps accuracy over 5km fiber point-to-point link. White Rabbit

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Clock signals can be de-jittered using a PLL/VCXO de-jitter circuitry on the node cards (< 1ps achievable) Triggers, sequences of pulses and gate signals can be de-jittered by sampling the signals with a de-jittered higher-frequency clock. –A deterministic signal delay of ½ the clock period will be added. Automatic skew adjustment using a clock loop phase compensation schema to compensate for different propagation delays. –up to 6 ns on the ATCA backplane from the central to the end slots Signal Integrity

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May EET TEST SETUP (FPGAS AND GIGABIT AURORA LINKS) ATCA Node cards Full-mesh able Fiber optic channel PCI timing master Up to 32 fiber optic channels

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Complete, implement and test the ATCA 3.7 e-Keying Setup a testbench for PTP timing over PCIe or SerialRapidIO including skew compensation Work out EET protocol compatibility issues between AMC/µTCA and ATCA platforms Performance tests: maximum frequency and minimum jitter/skew achievable on a standard backplane when using compensation Get feedback from the laboratories about their timing requirements! Future work

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Acknowledgments: This work has been carried out within the framework of the Contract of Association between the European Atomic Energy Community and "Instituto Superior Técnico" (IST). The views and opinions expressed herein do not necessarily reflect those of the European Commission. IST also received financial support from "Fundação para a Ciência e Tecnologia" in the frame of the Contract of Associated Laboratory. The PICMG (PCI Industrial Computer Manufacturers Group) is a consortium of companies who collaboratively develop open specifications for high performance telecommunications and industrial computing applications. THE END

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Ports: CLK1, CLK2 assigned, only CLK3 is user defined Lines:6 differential pairs Performance:Fair, 100 MHz, bussed Compatibility:Compatible Bussed clocks interface

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Ports: Timing and Data hub in slots 1 or 2 Lines: All nodes can receive or send 2 distinct timing signal Performance: Good; > 200 MHz, 1 GHz?; P2P Compatibility: Changes to ATCA 3.0 required Base interface

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Ports: Timing and Data lines share a fabric port Lines:All nodes receive or send up to 6 distinct timing signals Performance:Good; ~3 GHz P2P; <100 ps jitter Compatibility:Changes to ATCA 3.x required. Max data rate will not be achieved. Shared Fabric port

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Ports: Timing hub in slot 3 or 4; Ports 3 or 4 on node cards Lines:All nodes receive or send up to 8 distinct timing signals Performance:Good *; >3 GHz point to point (P2P); <100 ps jitter Compatibility: Addition of ATCA 3.7. Slot 3 dedicated to timing. Fabric star on hub slots * Can be further improved with the implementation of equal lenght,low crosstalk lines on new backplanes.

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Ports: Update channel ports redefined for timing Lines:Up to 10 additional, equal length, low crosstalk differential pairs Performance:Excellent; Low skew; Low jitter Compatibility:Not compatible in systems using the update channels Dedicated backplane lines

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Timing signals skew

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Signal integrity - triggers Triggers, sequences of pulses and gate signals can be de-jittered by sampling the signals with a de-jittered clock of the same domain. –Higher jitter than the sampling clock due to the addition of a flip-flop. –Sampling period must be less than the smallest trigger width. –A deterministic signal delay of ½ the clock period will be added. –Each clock domain signals must use one PLL.

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Base Interface specifies 10/100/1000BASE-T –uses 2 ports, 1000BASE-BX needs just one. –wastes board space as it requires PHY chips and magnetics. These drawbacks were recognized: –AMC.2 specifies up to two 1000BASE-BX interfaces (E1, E2). –ATCA 3.1 specifies 1000BASE-BX (and 10GBASE-BX4) on the fabric –A 10GBASE-KR ( Gbit/s, 64B/66B) (and a 40GBASE-KX4) fabric specification is in progress. A possible extension to the ATCA Base Interface specification would: –add e-keying for 1000BASE-BX, 10GBASE-KR and EET –thus, 2 distinct ports on base would be available (two x1, 4 pairs) –then, a mix of Ethernet, White Rabbit (WR) or other EET timing interface could coexist This extension allows the timing hub to be implemented on a ‘hybrid’ hub card: –which provide ‘reduced’ timing capability (one WR/EET port only on each node board) –that can be used on a dual-star only backplane –that can be used as complementary or substitute of a timing hub in slot 3 (and 4) –Node cards shall be aware of timing signals from base/slot 3/clock ports –This setup allows redundant setup (hybrid hub cards in slot 1 and 2) Timing signals over base interface

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May OptionDescriptionPort0Port1 110/100/1000BASE-TE 21000BASE-BXE BASE-BXE1E BASE-BX + EET (WR) (mBST) E1EET 510GBASE-KRE1 610GBASE-KRE1E2 710GBASE-KR + EET (WR) (mBST) E1EET Port Configuration Options

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May From PICMG® 3.0 R3.0 - AdvancedTCA® Base Specification: –6.6.2 Electrical design requirements ¶ 85 Boards must be capacitively coupled at the receiver interface to isolate transmitter and receiver common mode voltages. Since Fabric Interfaces are point-to-point, tristate capability is not typically available in the transmitters, hence the capacitive coupling. Currently supported protocol signals are DC balanced using 8B/10B or other encoding. As a result, capacitive coupling is possible. The value of the coupling capacitor and resistive terminations are protocol dependent and is specified in the respective PICMG® 3.0 subsidiary specifications. –Electrical requirements REQ 6.80 Fabric Interface receivers shall be capacitively coupled to the Backplane connection. REQ 6.81 Fabric Interface transmitters shall be direct coupled to the Backplane connection. Pulses, sequences of pulses or gates, contain signal frequencies near DC, ruling out the use of capacitors in the signal path. Solutions: –Provide tristate capability on timing transmitters. Correct DC coupling shall use similar logic family in both Hub and Node cards. Maximum frequency may be limited. –Modulate timing signals on a carrier frequency instead of base band. Fabric capacitive coupling

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May Drawbacks: –Additional modulator/demodulator device per line –Timing performance will likely be reduced. –Asynchronicity between distinct signals is lost (a phase error will happen) Encoders sample the original timing signal using a clock synchronous to the carrier clock. Candidates are Manchester encoding, 4b/5b and 8b/10b encoding: – – – – Modulated timing signals

Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, May While Manchester encoding is older it is not found often in gigabit implementations due to the low performance (halves the bandwidth). Manchester timing performance is better, but may be difficult to implement with standard devices. 8b/10b encoding reduces bandwidth by 20 % and can be found in various current gigabit implementations. While 8b/10b encoding provides a timing performance 5 times slower over Manchester, 8 distinct signals can be transmitted simultaneously per gigabit line. 4b/5b and 8b/10b encoding schemes allow error detection and control. Comparison of encoding methods Table 1 - Timing performance for some encoding schemes for bit rates up to 10 Gbit/s. (prospectively feasible to attain in ATCA fabric lines)