1. 17 th Real-Time Conference | Lisboa, 24-28 May 2010 An overview of the ATCA ® timing and synchronization resources for control and data acquisition J. Sousa 1, A.M. Fernandes, A.J.N. Batista, M. Correia, H. Fernandes, B.B. Carvalho, B. Gonçalves 1, C.A.F. Varandas 1 Member PICMG for Physics Committees 1 Instituto de Plasmas e Fusão Nuclear Instituto Superior Técnico, Lisbon, Portugal http://www.ipfn.ist.utl.pt

2. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 2 PICMG consortium is currently working on a extension of the ATCA family targeting Physics experiments I&C (xTCA) New hardware Specifications are being developed to: –Define additional Rear Transition Module connectivity (ARTM and uRTM) –Add additional Timing/Synchronization resources –Better the compatibility between µTCA and ATCA platforms The scope of this presentation is the timing extensions for the ATCA 3.x specification. The following slides will present: –An overview of the available Timing resources on ATCA –The current proposal for Timing on xTCA –Example of implementation on Hub and Node cards (AMC carrier) –Candidate timing network protocols for fabric/external connections –Signal integrity concerns –Test results for a synchronous timing network implementation Introduction

3. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 3 Timing overPortsLines availablePerformanceCompatibility Bussed Clock Interface CLK1, CLK2 assigned Only CLK3 may be used 6 diff pairsFair 100 MHz bussed Compatible Base interfaceTiming+Data hub in slots 1;2 All nodes receive/send 2 distinct timing signal Good ~1 GHz? P2P Changes to ATCA 3.0 required Shared Fabric port Timing+Data lines share a fabric port All nodes receive/send up to 6 distinct timing signals Good ~3 GHz P2P <100 ps jitter Changes to ATCA 3.x required Fabric star on hub slots Timing hub in slot 3;4 Ports 3;4 on node cards All nodes receive/send up to 8 distinct timing signals Good * ~3 GHz P2P <100 ps jitter Addition of ATCA 3.6 Dedicated backplane lines Update channel ports redefined for timing Up to 10 additional, equal length, low crosstalk diff pairs Excellent # Low skew Low jitter Not compatible if when update channels in use ATCA Timing Resources # When using equal lenght lines on backplanes and zero delay buffers * Can be further improved with the implementation of equal lenght,low crosstalk lines on new backplanes.

4. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 4 Goal: use current backplanes – compatibility with PICMG 3.x. Proposal: hub on slot 3 (+ 4) with new Link Type for timing signals. Hub can be used on dual-dual star and full-mesh backplanes. The timing hub will provide point-to-point and bidirectional links to each node card and implement the following functions: –local generation or distribution of timing signals to the node cards –route any timing signal from/to any node card (broad/multicast or P2P) The following types of timing signals are to be supported: Timing signals over fabric NameSignal typeDescription BSTBinary State Timingclock, trigger and gate type signals mBSTMultiplexed BSTsynchronous time multiplexed BST signals on a ‘n’- bit word (e.g. 8b/10b -> n = 8) EETEncoded Event and Timing Deterministic timing messages (sequence of words) modulated on a carrier clock. Carry trigger/event signals transport, clock skew correction, and time stamp

5. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 5 Dual dual-star backplane routing Logical Slot # 12345678910111213141516 Con.Ch # P2015 16-116-216-316-4 P2014 15-115-215-315-4 P2013 14-114-214-314-4 P2112 13-113-213-313-4 P2111 12-112-212-312-4 P2110 11-111-211-311-4 P219 10-110-210-310-4 P218 9-19-29-39-4 P227 8-18-28-38-4 P226 7-17-27-37-4 P225 6-16-26-36-4 P224 5-15-25-35-44-44-54-64-74-84-94-104-114-124-134-144-15 P223 4-14-24-33-33-43-53-63-73-83-93-103-113-123-133-143-15 P232 3-13-22-22-32-42-52-62-72-82-92-102-112-122-132-142-15 P231 2-11-11-21-31-41-51-61-71-81-91-101-111-121-131-141-15 SLOT-CHANNEL Row #Interface Designation J22/P22 Connector Pairs (Node board) abcdefgh 9Fabric Channel 3 BST_Tx2[3]BST_Rx2[3]BST_Tx3[3]BST_Rx3[3] 10 EET_Tx[3]EET_Rx[3]BST_Tx1[3]BST_Rx1[3] X4 channel 8 diff pairs

6. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 6 ValueDefinition 00h Reserved 01h PICMG 3.0 Base Interface 10/100/1000 BASE-T 02h PICMG 3.1 Ethernet Fabric Interface 03h PICMG 3.2 InfiniBand Fabric Interface 04h PICMG 3.3 StarFabric Fabric Interface 05h PICMG 3.4 PCI Express Fabric Interface 06h PICMG 3.5 Serial RapidlO Fabric Interface 07h PICMG 3.7 Timing over Fabric Interface 08h..EFh Reserved F0h..FEh OEM FFh Reserved e-Keying Type ExtensionDescription 0000b BST: High-Speed LVDS signalling (< 1 Gbit) (ANSI TIA/EIA-644 Standard 0001b BST: CML signalling (< 6.5 Gbit, 4.25 Gbit). 0010b-1011b Reserved 1100b-1111b OEM OptionSignalsPort 0 Port 1 Port 2 Port 3 1 EET X 2 mBST X 3 BST1,2,3 XXX 4 EET + BST1,2,3 XXXX 5 mBST + BST1,2,3 XXXX Port Configuration options up to 3 BST and 1 EET ports: Link TypeLink Extension mBST and EET signaling: adopt PICMG 3.5 (SRIO) PHY (1.25, 2.5, 3.125, 5 Gbaud)

7. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 7 Timing on AMC Carrier example

8. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 8 Example of Timing Hub

9. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 9 Criteria: time jitter and channel to channel skew performance Options: Proprietary (e.g. using FPGA) –Can be tailored for maximum synchronization performance –Higher level synchronization protocols (PTP) and link delay compensation (down to ps) can be straightforwardly added –Example: DESY system wide synchronization (μTCA) for XFEL. IEEE-1588 (PTP) over Ethernet –Synchronization independent of link delay to a degree of accuracy White Rabbit –PTP over Synchronous Ethernet (endpoints recover the COMM clock) –Includes line delay compensation PTP timing over PCIe or SerialRapidIO? –Lower message time jitter –Inherently synchronous, link delay compensation can be added EET candidate protocols

10. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 10 “This standard defines a protocol enabling precise synchronization of clocks in measurement and control systems. Networks shall support multicast messaging including but not limited to Ethernet”. Actual synchronization accuracies as low as 50 ns. Not ‘synchronous’. No fine delay adjustment. IEEE-1588 (PTP)

11. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 11 Uses hardware-assisted PTP for clock synchronization. –Synchronizes local clock with the master clock by measuring and compensating the coarse delay (using PTP) introduced by the link in multiples of 8 ns. –Fine delay adjustment is performed using a digital ‘Dual Mixer Time Difference’ which outputs a measure of the clock loop phase driving an adjustable delay element. All network nodes use the same physical layer clock (125 MHz), generated by the System Timing Master. Clock is encoded in the Ethernet carrier and recovered by the PLL in the PHY. Uses packet preemption and deterministic protocol for timing correction messages. –When timing messages arrives at a switch, the low-priority packet currently being routed is terminated so the HP packet can be sent out with minimal latency. Preliminary AMC protoype measurements: ± 350 ps, STD=80ps accuracy over 5km fiber point-to-point link. White Rabbit

12. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 12 Clock signals can be de-jittered using a PLL/VCXO de-jitter circuitry on the node cards (< 1ps achievable) Triggers, sequences of pulses and gate signals can be de-jittered by sampling the signals with a de-jittered higher-frequency clock. –A deterministic signal delay of ½ the clock period will be added. Automatic skew adjustment using a clock loop phase compensation schema to compensate for different propagation delays. –up to 6 ns on the ATCA backplane from the central to the end slots Signal Integrity

13. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 13 EET TEST SETUP (FPGAS AND GIGABIT AURORA LINKS) ATCA Node cards Full-mesh able Fiber optic channel PCI timing master Up to 32 fiber optic channels

14. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 14 Complete, implement and test the ATCA 3.7 e-Keying Setup a testbench for PTP timing over PCIe or SerialRapidIO including skew compensation Work out EET protocol compatibility issues between AMC/µTCA and ATCA platforms Performance tests: maximum frequency and minimum jitter/skew achievable on a standard backplane when using compensation Get feedback from the laboratories about their timing requirements! Future work

15. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 15 Acknowledgments: This work has been carried out within the framework of the Contract of Association between the European Atomic Energy Community and "Instituto Superior Técnico" (IST). The views and opinions expressed herein do not necessarily reflect those of the European Commission. IST also received financial support from "Fundação para a Ciência e Tecnologia" in the frame of the Contract of Associated Laboratory. The PICMG (PCI Industrial Computer Manufacturers Group) is a consortium of companies who collaboratively develop open specifications for high performance telecommunications and industrial computing applications. THE END

16. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 16 Ports: CLK1, CLK2 assigned, only CLK3 is user defined Lines:6 differential pairs Performance:Fair, 100 MHz, bussed Compatibility:Compatible Bussed clocks interface

17. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 17 Ports: Timing and Data hub in slots 1 or 2 Lines: All nodes can receive or send 2 distinct timing signal Performance: Good; > 200 MHz, 1 GHz?; P2P Compatibility: Changes to ATCA 3.0 required Base interface

18. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 18 Ports: Timing and Data lines share a fabric port Lines:All nodes receive or send up to 6 distinct timing signals Performance:Good; ~3 GHz P2P; <100 ps jitter Compatibility:Changes to ATCA 3.x required. Max data rate will not be achieved. Shared Fabric port

19. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 19 Ports: Timing hub in slot 3 or 4; Ports 3 or 4 on node cards Lines:All nodes receive or send up to 8 distinct timing signals Performance:Good *; >3 GHz point to point (P2P); <100 ps jitter Compatibility: Addition of ATCA 3.7. Slot 3 dedicated to timing. Fabric star on hub slots * Can be further improved with the implementation of equal lenght,low crosstalk lines on new backplanes.

20. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 20 Ports: Update channel ports redefined for timing Lines:Up to 10 additional, equal length, low crosstalk differential pairs Performance:Excellent; Low skew; Low jitter Compatibility:Not compatible in systems using the update channels Dedicated backplane lines

21. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 21 Timing signals skew

22. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 22 Signal integrity - triggers Triggers, sequences of pulses and gate signals can be de-jittered by sampling the signals with a de-jittered clock of the same domain. –Higher jitter than the sampling clock due to the addition of a flip-flop. –Sampling period must be less than the smallest trigger width. –A deterministic signal delay of ½ the clock period will be added. –Each clock domain signals must use one PLL.

23. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 23 Base Interface specifies 10/100/1000BASE-T –uses 2 ports, 1000BASE-BX needs just one. –wastes board space as it requires PHY chips and magnetics. These drawbacks were recognized: –AMC.2 specifies up to two 1000BASE-BX interfaces (E1, E2). –ATCA 3.1 specifies 1000BASE-BX (and 10GBASE-BX4) on the fabric –A 10GBASE-KR (10.3125 Gbit/s, 64B/66B) (and a 40GBASE-KX4) fabric specification is in progress. A possible extension to the ATCA Base Interface specification would: –add e-keying for 1000BASE-BX, 10GBASE-KR and EET –thus, 2 distinct ports on base would be available (two x1, 4 pairs) –then, a mix of Ethernet, White Rabbit (WR) or other EET timing interface could coexist This extension allows the timing hub to be implemented on a ‘hybrid’ hub card: –which provide ‘reduced’ timing capability (one WR/EET port only on each node board) –that can be used on a dual-star only backplane –that can be used as complementary or substitute of a timing hub in slot 3 (and 4) –Node cards shall be aware of timing signals from base/slot 3/clock ports –This setup allows redundant setup (hybrid hub cards in slot 1 and 2) Timing signals over base interface

24. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 24 OptionDescriptionPort0Port1 110/100/1000BASE-TE 21000BASE-BXE1 31000BASE-BXE1E2 41000BASE-BX + EET (WR) (mBST) E1EET 510GBASE-KRE1 610GBASE-KRE1E2 710GBASE-KR + EET (WR) (mBST) E1EET Port Configuration Options

25. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 25 From PICMG® 3.0 R3.0 - AdvancedTCA® Base Specification: –6.6.2 Electrical design requirements ¶ 85 Boards must be capacitively coupled at the receiver interface to isolate transmitter and receiver common mode voltages. Since Fabric Interfaces are point-to-point, tristate capability is not typically available in the transmitters, hence the capacitive coupling. Currently supported protocol signals are DC balanced using 8B/10B or other encoding. As a result, capacitive coupling is possible. The value of the coupling capacitor and resistive terminations are protocol dependent and is specified in the respective PICMG® 3.0 subsidiary specifications. –Electrical requirements REQ 6.80 Fabric Interface receivers shall be capacitively coupled to the Backplane connection. REQ 6.81 Fabric Interface transmitters shall be direct coupled to the Backplane connection. Pulses, sequences of pulses or gates, contain signal frequencies near DC, ruling out the use of capacitors in the signal path. Solutions: –Provide tristate capability on timing transmitters. Correct DC coupling shall use similar logic family in both Hub and Node cards. Maximum frequency may be limited. –Modulate timing signals on a carrier frequency instead of base band. Fabric capacitive coupling

26. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 26 Drawbacks: –Additional modulator/demodulator device per line –Timing performance will likely be reduced. –Asynchronicity between distinct signals is lost (a phase error will happen) Encoders sample the original timing signal using a clock synchronous to the carrier clock. Candidates are Manchester encoding, 4b/5b and 8b/10b encoding: –http://en.wikipedia.org/wiki/Manchester_encodinghttp://en.wikipedia.org/wiki/Manchester_encoding –http://en.wikipedia.org/wiki/Differential_Manchester_encodinghttp://en.wikipedia.org/wiki/Differential_Manchester_encoding –http://en.wikipedia.org/wiki/4B5Bhttp://en.wikipedia.org/wiki/4B5B –http://en.wikipedia.org/wiki/8B/10B_encodinghttp://en.wikipedia.org/wiki/8B/10B_encoding Modulated timing signals

27. Author’s name | Place, Month xx, 2007 | Event17 th Real-Time Conference | Lisboa, 24-28 May 2010 27 While Manchester encoding is older it is not found often in gigabit implementations due to the low performance (halves the bandwidth). Manchester timing performance is better, but may be difficult to implement with standard devices. 8b/10b encoding reduces bandwidth by 20 % and can be found in various current gigabit implementations. While 8b/10b encoding provides a timing performance 5 times slower over Manchester, 8 distinct signals can be transmitted simultaneously per gigabit line. 4b/5b and 8b/10b encoding schemes allow error detection and control. Comparison of encoding methods Table 1 - Timing performance for some encoding schemes for bit rates up to 10 Gbit/s. (prospectively feasible to attain in ATCA fabric lines)