1. ISOTDAQ International School Of Trigger and DAQ
Peter Jansweijer

2. ISOTDAQ International School Of Timing for DAQ
Peter Jansweijer

3. Timing for DAQ Outline What is time? Why is timing important?
Timing system considerations and fundamentals
Timing system examples
GNSS: Global Navigation Satellite System
TTC: Timing Trigger and Control
GBT: GigaBit Transceiver
WR: White Rabbit

4. What is time?

5. How to keep track of time
Calendar based on Sun, Moon and Stars
Sundials, water clocks
1656 Christiaan Huygens: Pendulum
1905 Einstein: time = 4th dimension
1920 Quartz oscillators
1950 atomic clocks
1967 definition of SI a) standard 1 sec based on Caesium atom
Currently: Optical atomic clocks
a) SI: Le système international d’unités

6. Picture: https://en.wikipedia.org/wiki/Earth%27s_orbit
Time Standards
Two widely used time standards:
Based on earth rotation
not uniform due to periodic changes and long term drifts
~1 sec/year (order < 10-8)
Based on atomic oscillations
Currently the closest approximation to a uniform time
~1 us/year (order < 10-14)
Picture:

7. Time standardization based on the rotation of the earth Solar time
varies due to eccentricity of the earth’s orbit around the sun
Greenwich Mean Time
1884 International Meridian Conference => longitude 0 is Greenwich meridian
GMT is Mean solar time at Greenwich Royal Observatory referring to noon (12:00:00)
UT1 = GMT referring to midnight (00:00:00)
Picture:

8. Time standardization based on the rotation of the earth
Meridian Telescope
Besançon Astronomical Observatory
Picture: courtesy Tjeerd Pinkert

9. based on the rotation of the earth
Time standardization
based on the rotation of the earth
Earth rotation suffers periodic changes and long term drifts
Ephemeris time was designed as an approach to a uniform time scale, to be freed from the effects of irregularity in the rotation of the earth
1956: astronomically determined ephemeris time:
« La seconde est la fraction 1/ ,9747 de l’année tropique pour 1900 janvier 0 à 12 heures de temps des éphémérides »
Difficult to work with:
Mean value of different second definitions
Corrections needed with respect to 1900

10. based on atomic oscillations
Time standardization
based on atomic oscillations
1967: atomic time
« L’unité de temps du Système international d’unités est la seconde définie dans les termes suivants: “La seconde est la durée de périodes de la radiation correspondant à la transition entre les deux niveaux hyperfins de l’état fondamental de l’atome de césium 133 ».
TAI (Temps Atomique International)
Is the average of ~400 clocks worldwide

11. Time standardization based on atomic oscillations
Picture: courtesy Tjeerd Pinkert
Caesium Clock at PTB, Braunschweig

12. Time standardization based on atomic oscillations
UTC (Temps Universel Coordonné) = related to TAI but occasionally leap seconds are introduced to correct for the difference between UTC and mean solar time (UT1).
Currently (February 2017): TAI – UTC = 37 second
Daily deviation of day length from SI day (86400 s)
+2 ms
Moving 365 day average
Cumulative deviation since introduction of leap seconds
Picture:

13. based on atomic oscillations
Time standardization
based on atomic oscillations
GPS Time (GPST)
Related to TAI
TAI - GPST = 19 seconds
No leap seconds
Try:
It shows the current time for various time standards

14. EPOCH A t0 reference point. For example: The birth of Christ
Buddha attained par nirvana (544 BC, Buddhist calendar)
November 17, 1858 = Modified Julian Day (MJD) 0
MJD starts at midnight 00:00
UNIX (00:00 January 1, 1970)
Try
Start of run (run number, event number)
Machine orbit

15. Why is time important?

16. Why is time important? Sorting event data (coincidences)
Track reconstruction
Positioning
𝑥= 1 𝜀 𝑟 .c.t
In a medium 1 ns » 20 cm
Telecommunications
Power grid
Financial systems
Picture:
Picture :
Picture :

17. Track reconstruction examples
Picture ATLAS Experiment website:

18. Track reconstruction examples
640 strings
18 DOM/string
DOMs
PMTs
Volume: ~5 km3
Cherenkov m n
DOMs in the deep sea at 3-5 km depth

19. Track reconstruction examples
Picture University of Nova Gorica website:
Very high energy g-ray observatory
Two arrays of 100 and 20 telescopes
Picture CTA website:

20. Picture SKA website: https://www.skatelescope.org/
Phase array example
Picture SKA website:

21. Timing system considerations and fundamentals

22. What constraints will influence your choice for a timing system?
Relative versus Absolute time
Time resolution and precision
Location
GPS visibility
Radiation
Use radiation tolerant electronics
Power
Thermal constraints, supply constraints
Size / Mass
area constraints, avoid scattering
Reliability
Space weather

23. Two approaches in DAQ systems
Fundamentals
Two approaches in DAQ systems
frontend & de-randomize
Fixed latency, timestamp later
Why is the first one best?
Frontend
Counting Room
Frontend
Counting Room

24. Why timestamp @ frontend is best
Fundamentals
Why frontend is best
Distribute frequency
Average clock edges
Added single shot channel jitter
Distributing frequency = Syntonization:
“The adjustment of two electronic circuits or devices in terms of frequency”

25. Timing system Examples

26. A Timing System should deliver:
A stable clock…
… which is phase aligned
Deliver a t0 reference
Frequency
Time

27. Global Navigation Satellite System (GNSS)

28. GNSS ГЛОбальная НАвигационная Спутниковая Система (GLONASS, RU)
Global Positioning System (GPS, US)
ГЛОбальная НАвигационная Спутниковая Система (GLONASS, RU)
Galileo (EU)
BeiDou navigation satellite System (Compass/BeiDou-2, CN, 2020)

29. GNSS At least 24 satellites for global coverage
Always 4 satellites visible
Orbital data message superimposed on a code that serves as a time reference (based on satellites atomic clock)
Trilateration
Location and time
GPST
accuracy ~ 100 ns
Picture:

30. The Timing Trigger and Control system
(TTC)

31. TTC Designed for LHC experiments (1995~2007)
Customized to LHC timing 7 TeV):
Bunch clock: 40,07897 MHz a)
Orbit: 11,2455 KHz a) (LHC circumference b) = m and C = m/s)
Orbit contains 40,07897e6 / 11,2455e3 = bunches a) b)

32. TTC distributes: LHC clock L1-trigger
with appropriate phase relative to LHC bunch structure (delays: particle time-of-flight / signal propagation)
Coarse deskew (16 bunches)
Fine deskew 240 steps of 104 ps
L1-trigger
Bunch Counter (increment on LHC clock)
Event Counter (increment on L1A)
Broadcast or individual addressed signals
Hamming code => error detection (detect 2, correct 1 error)

33. TTC wrap-up Frequency: LHC Clock Epoch: Start of run
run_number, event_number (+ bunch number)
Radiation tolerant
Hamming code
Radtol TTCrx chip and QPLL

34. GigaBit Transceiver (GBT)

35. GBT Designed for High Energy Physics (LHC) experiments (2007~2017)
Combining:
Readout (DAQ) + Timing (TTC) + Slow control (SC)
Picture:

36. GBT Radiation tolerant Bidirectional bandwidth 3,2 – 4,48 Gbps
Forward Error Correction (in standard mode)
Correct single bit and burst errors
Triple Modular Redundancy configuration registers
Bidirectional bandwidth 3,2 – 4,48 Gbps
Chipset:
GBTx (Transceiver)
GBTIA (Trans Impedance Amplifier)
GBLD (Laser Driver)
GBT-SCA (Slow Control Adapter)

37. GBTx Constant latency 8 reference clocks are programmable in b)
enables the GBTx to be used in clock synchronous trigger systems and for precise TTC distribution a)
synchronous with LHC bunch crossing reference
stable phase relationship
8 reference clocks are programmable in b)
Phase: 48,8 ps resolution (LHC clock / 512 steps)
Frequency: 40, and 320 MHz
Chapter 4
Chapter 12

38. Clock source and distribution
GBT Transceiver modes
Clock source and distribution
Detector
Counting Room
Pictures:

39. GBT wrap-up Frequency: LHC Clock Epoch: Start of run
run_number, event_number (+bunch number)
Combining readout, timing, detector control
Radiation tolerant
Read Solomon FEC

40. Timing over Ethernet

41. Network Time Protocol (NTP)
Since 1985
UTC millisecond -> microsecond precision
UDP port 123
NTP Client regularly polls three or more NTP servers
Client
Server t1 t4 t2 t3
time
time offset q
+ or -
Provided that:
Subtract equations => Time offset:
Add equations => Round-trip delay: 41

42. Precision Time Protocol (IEEE 1588)
Since 2002 (v1), (v2)
Link delay evaluated by measuring and exchanging frames with tx/rx hardware timestamps
Sub microsecond precision
Frame-based (Multicast Ethertype 0x88F7 or UDP port 320 and IP multicast)
Synchronizes local clock with master clock.
(v2)

43. White Rabbit

44. High performance timing over Ethernet
White Rabbit
High performance timing over Ethernet
Bandwidth: 1 Gbps
Single fiber medium
Up to 10 km links
WR Switch: 18 ports
Allows non-WR Devices
Ethernet features (VLAN) & protocols (SNMP)

45. High performance timing over Ethernet
White Rabbit
High performance timing over Ethernet
Two separate services
(enhancements to Ethernet) provided by WR:
Synchronization: accuracy better than 1 ns precision (tens of ps sdev skew max)
Deterministic, reliable and low-latency Control Data delivery

46. White Rabbit Switch Central element of WR network
Central element of WR network
BASE-BX10 ports
Open design (H/W and S/W)
Commercially available

47. White Rabbit Node example 1: SPEC
used in Lab 8
(without timing part)
FMC-based Hardware Kit
All carrier cards are equipped with a White Rabbit port.
Mezzanines can use the accurate clock signal and “TAI” (synchronous sampling clock, trigger, time tag, …).
Starting kit:

48. High performance timing over Ethernet
kA = kB
Syntonization
Synchronous Ethernet
bA = bB
synchronisation
Precision Time Protocol (PTP)

49. Layer 1 Syntonization
Syntonization = “The adjustment of two electronic circuits or devices in terms of frequency”
All network devices use the same physical layer clock.
Clock is encoded in the Ethernet carrier and recovered by the receiver chip.
Phase detection allows sub-ns delay measurement

50. Precision Time Protocol (IEEE 1588)
Frame-based synchronization protocol.
Synchronizes local clock with master clock.
Link delay evaluated by measuring and exchanging frames with tx/rx timestamps

51. Digital Dual Mixer Time Difference
(DDMTD)

52. Using the DDMTD as phase detector and phase shifter

53. White Rabbit PTP Core
T-Sense /
Unique ID
T-Sense /
Unique ID

54. White Rabbit wrap-up Frequency: Any master clock Time: TAI (UTC)
distributed using Layer-1 based synchronisation
delivers phase aligned reference clock
Time: TAI (UTC)
Pulse Per Second (PPS)
Epoch: UNIX (00:00 January 1, 1970)
May be used instead of GPS
at locations where satellite line of sight is obstructed
when sub ns precision is needed

55. 14-16 March, 2016 hosted by Nikhef
9th WR workshop
14-16 March, 2016 hosted by Nikhef 66
Participants 30
Institutes
Universities
Companies 9
Countries
Worldwide
See also:

56. Future: PTP Standardization
Jan 2017: Draft version of IEEE1588 (PTP v3)
Mid 2018: Publish IEEE
White Rabbit => High Accuracy profile
Annex O: Layer-1 based synchronization performance enhancement
Annex P: Sub-ns synchronization using the High Accuracy Default PTP Profile
Annex Q: Relative Calibration Procedures
More about the standardization procedure:

57. Thank you Concluding: There is more to time than day and night
And many thanks to:
Henk Peek, Tjeerd Pinkert, Andrea Borga
all WR developers / contributors
(also for re-using many of the White Rabbit slides)

58. Backup Slides

59. TTC
“The overall TTC system architecture provides for the distribution of synchronous timing, level-1 trigger, and broadcast and individually- addressed control signals, to electronics controllers with the appropriate phase relative to the LHC bunch structure, taking account of the different delays due to particle time-of-flight and signal propagation. Within each trigger distribution zone, the signals can be broadcast from a single laser source to several hundred destinations over a passive network composed of a hierarchy of optical tree couplers. “ a) a)

60. TTC BiPhase Mark @ 160,32 MBaud
Picture:

61. TTC Broadcast commands:
Short-format Broadcast signals
Synchronous to machine orbit and deskewed
Bunch Counter Reset, Event Counter Reset
Asynchronous
Long-format Individual addressed slow control signals
Setting TTC Receiver internal registers:
control and configuration
controlling deskew coarse and fine delay
8-bit Trigger Type + 24-bit Event Number (for check purposes)

62. Picture: http://ttc.web.cern.ch/TTC/AlbuquerqueNSS97.pdf
TTC system overview
TTCvi
TTCex, TTCvx
TTCoc
TTCrx chip,
TTCrq mezzanine
Picture:

63. (VME bus interface module)
TTC modules
TTCvi
(VME bus interface module)

64. (laser transmitter module)
TTC modules
TTCex or TTCvx
(laser transmitter module)
TTCoc (1:32 optical tree coupler)

65. TTC modules Too much clock jitter… …Upgraded with QPLL
TTCrx chip (TTC receiver)
TTCrm
(TTC receiver mezzanine)
TTCrq
(TTC receiver mezzanine
With QPLL)

66. GBT
Picture:

67. GBT frame format User data = 80 bits @ 25 ns = 3,2 Gbps
without FEC: ns = 4,48 Gbps
H(3:0)
IC(1:0)
EC(1:0)
D(79:64)
D(63:48)
D(47:32)
D(31:16)
D(15:0)
FEC(31:16)
FEC(15:0)
Bits
Purpose 4
Header (H) 2
Internal Conrtol (IC)
External Conrtol (EC) 80
User Data (D) + 32
Forward Error Correction (FEC)
120
GBT 25 ns LHC clock

68. How to implement WR in your design

69. How to implement White Rabbit in your design
FPGA
SFP
PHY
DATA
(1 Gbps)
Unique-ID
(MAC-addr)
(optional)
GTREFCLK
SPI
Your own
stuff
DAC
VCXO
125 MHz
CLK125M_PLLREF
REF clock
generator
CLK_DMTD
CLK20M_VCXO
DAC
VCXO
PLL
20+ MHz
DMTD clock
generator
Timing
(TAI[ns],REFCLK)

70. Many use cases… 1000 Km link in Finland (Anders Wallin)
100 times longer than the original 10 Km specification
2 ns error over 60 days

71. VLBI

72. Autonomous driving car Pictures: Jeroen Koelemeij
SuperGPS
Optical methods to back up GNSS timing via fiber
Next-generation positioning (optical/radio)
Autonomous driving car
Pictures: Jeroen Koelemeij
SuperGPS

73. Metrology institutes discover potentials
international optical clock
Comparisons using optical fibers
See also presentations at the
Third International VLBI Technology Workshop

74. Other White Rabbit Node examples:
CUTE-WR (LHAASO)
Central Logic Board (KM3NeT)
CRIO-WR (CERN)
SVEC (CERN)
SPEXI (CERN)

75. Open Hardware Repository
Estimation: White Rabbit up to now (2017):
Over 100 man years of work!
Commercial
Non-commercial
Open
Winning combination.
Best of both worlds.
Whole support burden falls on developers. Not scalable.
Proprietary
Vendor lock-in.
Dedicated
non-reusable projects.

76. White Rabbit PTP Core in detail
T-Sense /
Unique ID
PHY:
Interface to the physical network, with deterministic phase relationship between gigabit clock and system clock

77. White Rabbit PTP Core in detail
T-Sense /
Unique ID
Endpoint:
Ethernet MAC, with precision time stamping capabilities of in- and out-bound Ethernet frames

78. White Rabbit PTP Core in detail
T-Sense /
Unique ID
Other
packets
Redirector:
Redirects all PTP Ethernet packets to the Mini-NIC
Receives the payload of the PTP packets and stores payload data in memory

79. White Rabbit PTP Core in detail
T-Sense /
Unique ID
Embedded Processor system
(LatticeMico32 + Bus interface + Memory)
running a PTP-daemon

80. White Rabbit PTP Core in detail
T-Sense /
Unique ID
Now the CPU has “knowledge” of time.
It can adjust the local time and update the proper TAI time and control the Pulse Per Second (PPS) generation

81. White Rabbit PTP Core in detail
The local oscillator that is used for TX is locked onto the RX receive clock. A PLL is made using Digital Dual Mixer Time Difference (DDMTD) technique and software on the LatticeMico32 (hence Soft-PLL) to tune the oscillator.
DDMTD allows for < 1 ns phase adjustment under control of the LatticeMico32 => a result of calculations of the round trip delay (Timing Synchro)
T-Sense /
Unique ID

82. White Rabbit PTP Core in detail
T-Sense /
Unique ID
1-Wire is used to fetch a unique MAC address UART is used for debug only