ISOTDAQ International School Of Trigger and DAQ Peter Jansweijer

ISOTDAQ International School Of Timing for DAQ Peter Jansweijer

Timing for DAQ Outline What is time? Why is timing important? Timing system considerations and fundamentals Timing system examples GNSS: Global Navigation Satellite System TTC: Timing Trigger and Control GBT: GigaBit Transceiver WR: White Rabbit

What is time?

How to keep track of time https://www.nat.vu.nl/en/Images/Tjeerd_Pinkert_tcm208-690957.pdf Calendar based on Sun, Moon and Stars Sundials, water clocks 1656 Christiaan Huygens: Pendulum 1905 Einstein: time = 4th dimension 1920 Quartz oscillators 1950 atomic clocks 1967 definition of SI a) standard 1 sec based on Caesium atom Currently: Optical atomic clocks a) SI: Le système international d’unités

Picture: https://en.wikipedia.org/wiki/Earth%27s_orbit Time Standards http://www.cv.nrao.edu/~rfisher/Ephemerides/times.html Two widely used time standards: Based on earth rotation not uniform due to periodic changes and long term drifts ~1 sec/year (order < 10-8) Based on atomic oscillations Currently the closest approximation to a uniform time ~1 us/year (order < 10-14) Picture: https://en.wikipedia.org/wiki/Earth%27s_orbit

Time standardization based on the rotation of the earth Solar time varies due to eccentricity of the earth’s orbit around the sun Greenwich Mean Time 1884 International Meridian Conference => longitude 0 is Greenwich meridian GMT is Mean solar time at Greenwich Royal Observatory referring to noon (12:00:00) UT1 = GMT referring to midnight (00:00:00) Picture: https://en.wikipedia.org/wiki/Equation_of_time

Time standardization based on the rotation of the earth Meridian Telescope Besançon Astronomical Observatory Picture: courtesy Tjeerd Pinkert

based on the rotation of the earth Time standardization based on the rotation of the earth Earth rotation suffers periodic changes and long term drifts Ephemeris time was designed as an approach to a uniform time scale, to be freed from the effects of irregularity in the rotation of the earth 1956: astronomically determined ephemeris time: « La seconde est la fraction 1/31 556 925,9747 de l’année tropique pour 1900 janvier 0 à 12 heures de temps des éphémérides » Difficult to work with: Mean value of different second definitions Corrections needed with respect to 1900

based on atomic oscillations Time standardization based on atomic oscillations 1967: atomic time « L’unité de temps du Système international d’unités est la seconde définie dans les termes suivants: “La seconde est la durée de 9 192 631 770 périodes de la radiation correspondant à la transition entre les deux niveaux hyperfins de l’état fondamental de l’atome de césium 133 ». TAI (Temps Atomique International) Is the average of ~400 clocks worldwide

Time standardization based on atomic oscillations Picture: courtesy Tjeerd Pinkert Caesium Clock at PTB, Braunschweig

Time standardization based on atomic oscillations UTC (Temps Universel Coordonné) = related to TAI but occasionally leap seconds are introduced to correct for the difference between UTC and mean solar time (UT1). Currently (February 2017): TAI – UTC = 37 second Daily deviation of day length from SI day (86400 s) +2 ms Moving 365 day average Cumulative deviation since introduction of leap seconds Picture: https://en.wikipedia.org/wiki/Leap_second

based on atomic oscillations Time standardization based on atomic oscillations GPS Time (GPST) Related to TAI TAI - GPST = 19 seconds No leap seconds Try: http://www.leapsecond.com/java/gpsclock.htm It shows the current time for various time standards

EPOCH A t0 reference point. For example: The birth of Christ Buddha attained par nirvana (544 BC, Buddhist calendar) November 17, 1858 = Modified Julian Day (MJD) 0 MJD starts at midnight 00:00 UNIX (00:00 January 1, 1970) Try http://www.epochconverter.com/ Start of run (run number, event number) Machine orbit

Why is time important?

Why is time important? Sorting event data (coincidences) Track reconstruction Positioning 𝑥= 1 𝜀 𝑟 .c.t In a medium 1 ns » 20 cm Telecommunications Power grid Financial systems Picture: https://www.national.co.uk/ Picture : http://leica-geosystems.com/ Picture : http://content.time.com/

Track reconstruction examples Picture ATLAS Experiment website: http://atlasexperiment.org/photos/events-collision-proton.html http://cdsweb.cern.ch/record/1459502

Track reconstruction examples 640 strings 18 DOM/string 11.520 DOMs 357.120 PMTs Volume: ~5 km3 Cherenkov m n 11.520 DOMs in the deep sea at 3-5 km depth

Track reconstruction examples Picture University of Nova Gorica website: http://www.ung.si/en/research/laboratory-for-astroparticle-physics/projects/cta/ Very high energy g-ray observatory Two arrays of 100 and 20 telescopes Picture CTA website: https://www.cta-observatory.org/

Picture SKA website: https://www.skatelescope.org/ Phase array example Picture SKA website: https://www.skatelescope.org/

Timing system considerations and fundamentals

What constraints will influence your choice for a timing system? Relative versus Absolute time Time resolution and precision Location GPS visibility Radiation Use radiation tolerant electronics Power Thermal constraints, supply constraints Size / Mass area constraints, avoid scattering Reliability Space weather

Two approaches in DAQ systems Fundamentals Two approaches in DAQ systems Timestamp/tag @ frontend & de-randomize Fixed latency, timestamp later Why is the first one best? Frontend Counting Room Frontend Counting Room

Why timestamp @ frontend is best Fundamentals Why timestamp @ frontend is best Distribute frequency Average clock edges Added single shot channel jitter Distributing frequency = Syntonization: “The adjustment of two electronic circuits or devices in terms of frequency”

Timing system Examples

A Timing System should deliver: A stable clock… … which is phase aligned Deliver a t0 reference Frequency Time

Global Navigation Satellite System (GNSS)

GNSS ГЛОбальная НАвигационная Спутниковая Система (GLONASS, RU) Global Positioning System (GPS, US) ГЛОбальная НАвигационная Спутниковая Система (GLONASS, RU) Galileo (EU) BeiDou navigation satellite System (Compass/BeiDou-2, CN, 2020)

GNSS At least 24 satellites for global coverage Always 4 satellites visible Orbital data message superimposed on a code that serves as a time reference (based on satellites atomic clock) Trilateration Location and time GPST accuracy ~ 100 ns Picture: https://openclipart.org/detail/191659/gps-satellites-trilateration

The Timing Trigger and Control system (TTC) http://ttc.web.cern.ch/ttc/

TTC Designed for LHC experiments (1995~2007) Customized to LHC timing (numbers @ 7 TeV): Bunch clock: 40,07897 MHz a) Orbit: 11,2455 KHz a) (LHC circumference b) = 26 659 m and C = 299792458 m/s) Orbit contains 40,07897e6 / 11,2455e3 = 3564 bunches a) http://www.hephy.at/project/cms/trigger/globalTrigger/trans/MEDIEN/Publications/CMSdetector_JINST/Info_CMS_JINST/TTC-Taylor_LECC02.pdf b) https://project-physicsteaching.web.cern.ch/project-physicsteaching/english/brochures/lhc-guide.pdf

TTC distributes: LHC clock L1-trigger with appropriate phase relative to LHC bunch structure (delays: particle time-of-flight / signal propagation) Coarse deskew (16 bunches) Fine deskew 240 steps of 104 ps L1-trigger Bunch Counter (increment on LHC clock) Event Counter (increment on L1A) Broadcast or individual addressed signals Hamming code => error detection (detect 2, correct 1 error)

TTC wrap-up Frequency: LHC Clock Epoch: Start of run run_number, event_number (+ bunch number) Radiation tolerant Hamming code Radtol TTCrx chip and QPLL

GigaBit Transceiver (GBT) https://cds.cern.ch/record/1091474/files/p332.pdf https://espace.cern.ch/GBT-Project/default.aspx https://espace.cern.ch/GBT-Project/GBTX/Manuals/gbtxManual.pdf

GBT Designed for High Energy Physics (LHC) experiments (2007~2017) Combining: Readout (DAQ) + Timing (TTC) + Slow control (SC) Picture: https://espace.cern.ch/GBT-Project/GBTX/Manuals/gbtxManual.pdf

GBT Radiation tolerant Bidirectional bandwidth 3,2 – 4,48 Gbps Forward Error Correction (in standard mode) Correct single bit and burst errors Triple Modular Redundancy configuration registers Bidirectional bandwidth 3,2 – 4,48 Gbps Chipset: GBTx (Transceiver) GBTIA (Trans Impedance Amplifier) GBLD (Laser Driver) GBT-SCA (Slow Control Adapter)

GBTx Constant latency 8 reference clocks are programmable in b) enables the GBTx to be used in clock synchronous trigger systems and for precise TTC distribution a) synchronous with LHC bunch crossing reference stable phase relationship 8 reference clocks are programmable in b) Phase: 48,8 ps resolution (LHC clock / 512 steps) Frequency: 40, 80 160 and 320 MHz https://espace.cern.ch/GBT-Project/GBTX/Manuals/gbtxManual.pdf Chapter 4 https://espace.cern.ch/GBT-Project/GBTX/Manuals/gbtxManual.pdf Chapter 12

Clock source and distribution GBT Transceiver modes Clock source and distribution Detector Counting Room Pictures: https://espace.cern.ch/GBT-Project/GBTX/Manuals/gbtxManual.pdf

GBT wrap-up Frequency: LHC Clock Epoch: Start of run run_number, event_number (+bunch number) Combining readout, timing, detector control Radiation tolerant Read Solomon FEC

Timing over Ethernet

Network Time Protocol (NTP) Since 1985 UTC millisecond -> microsecond precision UDP port 123 NTP Client regularly polls three or more NTP servers   Client Server t1 t4 t2 t3 time time offset q + or - Provided that:   Subtract equations => Time offset:   Add equations => Round-trip delay:   41

Precision Time Protocol (IEEE 1588) Since 2002 (v1), 2008 (v2) Link delay evaluated by measuring and exchanging frames with tx/rx hardware timestamps Sub microsecond precision Frame-based (Multicast 01- 1-19-00-00-00 Ethertype 0x88F7 or UDP port 320 and IP multicast) Synchronizes local clock with master clock. 1588-2008 (v2)

White Rabbit http://www.ohwr.org/projects/white-rabbit/wiki

High performance timing over Ethernet White Rabbit High performance timing over Ethernet Bandwidth: 1 Gbps Single fiber medium Up to 10 km links WR Switch: 18 ports Allows non-WR Devices Ethernet features (VLAN) & protocols (SNMP)

High performance timing over Ethernet White Rabbit High performance timing over Ethernet Two separate services (enhancements to Ethernet) provided by WR: Synchronization: accuracy better than 1 ns precision (tens of ps sdev skew max) Deterministic, reliable and low-latency Control Data delivery

White Rabbit Switch Central element of WR network http://www.ohwr.org/projects/white-rabbit/wiki/Switch Central element of WR network 18 1000BASE-BX10 ports Open design (H/W and S/W) Commercially available

White Rabbit Node example 1: SPEC http://www.ohwr.org/projects/spec/wiki used in Lab 8 (without timing part) FMC-based Hardware Kit All carrier cards are equipped with a White Rabbit port. Mezzanines can use the accurate clock signal and “TAI” (synchronous sampling clock, trigger, time tag, …). Starting kit: http://www.ohwr.org/projects/wr-starting-kit/wiki

High performance timing over Ethernet kA = kB Syntonization Synchronous Ethernet bA = bB synchronisation Precision Time Protocol (PTP)

Layer 1 Syntonization Syntonization = “The adjustment of two electronic circuits or devices in terms of frequency” All network devices use the same physical layer clock. Clock is encoded in the Ethernet carrier and recovered by the receiver chip. Phase detection allows sub-ns delay measurement

Precision Time Protocol (IEEE 1588) Frame-based synchronization protocol. Synchronizes local clock with master clock. Link delay evaluated by measuring and exchanging frames with tx/rx timestamps

Digital Dual Mixer Time Difference (DDMTD)

Using the DDMTD as phase detector and phase shifter

White Rabbit PTP Core http://www.ohwr.org/projects/wr-cores/wiki/Wrpc_core T-Sense / Unique ID T-Sense / Unique ID

White Rabbit wrap-up Frequency: Any master clock Time: TAI (UTC) distributed using Layer-1 based synchronisation delivers phase aligned reference clock Time: TAI (UTC) Pulse Per Second (PPS) Epoch: UNIX (00:00 January 1, 1970) May be used instead of GPS at locations where satellite line of sight is obstructed when sub ns precision is needed

14-16 March, 2016 hosted by Nikhef 9th WR workshop 14-16 March, 2016 hosted by Nikhef 66 Participants 30 Institutes Universities Companies 9 Countries Worldwide See also: http://www.ohwr.org/projects/white-rabbit/wiki/Workshops

Future: PTP Standardization Jan 2017: Draft version of IEEE1588 (PTP v3) Mid 2018: Publish IEEE1588-2018 White Rabbit => High Accuracy profile Annex O: Layer-1 based synchronization performance enhancement Annex P: Sub-ns synchronization using the High Accuracy Default PTP Profile Annex Q: Relative Calibration Procedures More about the standardization procedure: http://www.ohwr.org/attachments/4249/WRworkshop-P1588-HA.pdf

Thank you Concluding: There is more to time than day and night And many thanks to: Henk Peek, Tjeerd Pinkert, Andrea Borga all WR developers / contributors (also for re-using many of the White Rabbit slides)

Backup Slides

TTC “The overall TTC system architecture provides for the distribution of synchronous timing, level-1 trigger, and broadcast and individually- addressed control signals, to electronics controllers with the appropriate phase relative to the LHC bunch structure, taking account of the different delays due to particle time-of-flight and signal propagation. Within each trigger distribution zone, the signals can be broadcast from a single laser source to several hundred destinations over a passive network composed of a hierarchy of optical tree couplers. “ a) a) http://ttc.web.cern.ch/TTC/intro.html

TTC BiPhase Mark @ 160,32 MBaud Picture: http://ttc.web.cern.ch/TTC/AlbuquerqueNSS97.pdf

TTC Broadcast commands: Short-format Broadcast signals Synchronous to machine orbit and deskewed Bunch Counter Reset, Event Counter Reset Asynchronous Long-format Individual addressed slow control signals Setting TTC Receiver internal registers: control and configuration controlling deskew coarse and fine delay 8-bit Trigger Type + 24-bit Event Number (for check purposes)

Picture: http://ttc.web.cern.ch/TTC/AlbuquerqueNSS97.pdf TTC system overview TTCvi TTCex, TTCvx TTCoc TTCrx chip, TTCrq mezzanine Picture: http://ttc.web.cern.ch/TTC/AlbuquerqueNSS97.pdf

(VME bus interface module) TTC modules TTCvi (VME bus interface module)

(laser transmitter module) TTC modules TTCex or TTCvx (laser transmitter module) TTCoc (1:32 optical tree coupler)

TTC modules Too much clock jitter… …Upgraded with QPLL TTCrx chip (TTC receiver) TTCrm (TTC receiver mezzanine) TTCrq (TTC receiver mezzanine With QPLL)

GBT Picture: https://espace.cern.ch/GBT-Project/GBTX/Manuals/gbtxManual.pdf

GBT frame format User data = 80 bits @ 25 ns = 3,2 Gbps without FEC: 112 bits @ 25 ns = 4,48 Gbps H(3:0) IC(1:0) EC(1:0) D(79:64) D(63:48) D(47:32) D(31:16) D(15:0) FEC(31:16) FEC(15:0) Bits Purpose 4 Header (H) 2 Internal Conrtol (IC) External Conrtol (EC) 80 User Data (D) + 32 Forward Error Correction (FEC) 120 GBT Frame @ 25 ns LHC clock

How to implement WR in your design

How to implement White Rabbit in your design FPGA SFP PHY DATA (1 Gbps) Unique-ID (MAC-addr) (optional) GTREFCLK SPI Your own stuff DAC VCXO 125 MHz CLK125M_PLLREF REF clock generator CLK_DMTD CLK20M_VCXO DAC VCXO PLL 20+ MHz DMTD clock generator Timing (TAI[ns],REFCLK) http://www.ohwr.org/projects/white-rabbit/wiki/WRReferenceDesign

Many use cases… 1000 Km link in Finland (Anders Wallin) 100 times longer than the original 10 Km specification 2 ns error over 60 days

VLBI

Autonomous driving car Pictures: Jeroen Koelemeij SuperGPS Optical methods to back up GNSS timing via fiber Next-generation positioning (optical/radio) Autonomous driving car Pictures: Jeroen Koelemeij http://www.ohwr.org/attachments/1753/Jeroen-superGPS.pptx SuperGPS

Metrology institutes discover potentials international optical clock Comparisons using optical fibers See also presentations at the Third International VLBI Technology Workshop

Other White Rabbit Node examples: CUTE-WR (LHAASO) Central Logic Board (KM3NeT) CRIO-WR (CERN) SVEC (CERN) SPEXI (CERN)

Open Hardware Repository Estimation: White Rabbit up to now (2017): Over 100 man years of work! http://www.ohwr.org/ Commercial Non-commercial Open Winning combination. Best of both worlds. Whole support burden falls on developers. Not scalable. Proprietary Vendor lock-in. Dedicated non-reusable projects.

White Rabbit PTP Core in detail T-Sense / Unique ID PHY: Interface to the physical network, with deterministic phase relationship between gigabit clock and system clock

White Rabbit PTP Core in detail T-Sense / Unique ID Endpoint: Ethernet MAC, with precision time stamping capabilities of in- and out-bound Ethernet frames

White Rabbit PTP Core in detail T-Sense / Unique ID Other packets Redirector: Redirects all PTP Ethernet packets to the Mini-NIC Receives the payload of the PTP packets and stores payload data in memory

White Rabbit PTP Core in detail T-Sense / Unique ID Embedded Processor system (LatticeMico32 + Bus interface + Memory) running a PTP-daemon

White Rabbit PTP Core in detail T-Sense / Unique ID Now the CPU has “knowledge” of time. It can adjust the local time and update the proper TAI time and control the Pulse Per Second (PPS) generation

White Rabbit PTP Core in detail The local oscillator that is used for TX is locked onto the RX receive clock. A PLL is made using Digital Dual Mixer Time Difference (DDMTD) technique and software on the LatticeMico32 (hence Soft-PLL) to tune the oscillator. DDMTD allows for < 1 ns phase adjustment under control of the LatticeMico32 => a result of calculations of the round trip delay (Timing Synchro) T-Sense / Unique ID

White Rabbit PTP Core in detail T-Sense / Unique ID 1-Wire is used to fetch a unique MAC address UART is used for debug only