Clock Distribution for IceCube Gerald. Przybylski Lawrence Berkeley National Laboratory, Design Review, September 16,

History string 18 implementation: o Rubidium module slaved to GPS Clock o fan-out port for each domcom card. string 21 implementation: + OCXO GPS clock, Symmetricom ET Passive fan-out to each DOM Hub (DSB Card) + Sub-nanosecond skew and jitter demonstrated + Simple and Reliable, but not scalable

MCU Requirements Straightforward; conceptually very simple 5ns absolute accuracy (skew and jitter), within the IceCube counting house > Across all DOM Hubs (at DOR cards) > Fixed and stable offset from Universal Time, Coordinated (UTC) > Based on Scattering Length in the ice Distribute 10 MHz, 1Hz and T ime V alue S tring Free from Metastable states/events; no glitches Measurable and Verifiable Single driver per output-port; no shared drivers Robustness requirements in ERD x Mainly dealing with satellite drop-out and loss of availability x Also dealing with tracking multiple GPS clocks Phase accuracy: 0.4ns at fan-out, 0.7ns at DSB, and 1.0 ns at DOR

IceCube + IceTop + AMANDA IceTop same as IceCube IceCube to AMANDA *** Zeuthen/Wuppertal; Install 05/06; Holger et. al. *** Fiberoptic transmitter driven by GPS clock Fiberoptic receiver drives TWR (GPS4TWR) Autonomous from the MCU Clock Fan-Out subsystem 10 MHz BNC, 1Hz BNC, IRIG-B BNC 10 ns precision with respect to IceCube time AMANDA to IceCube *** Same players Trigger system signals over Fiberoptic link to IceCube counting house Depends on DOM Main Board(s) on a “String 81” Justification: “We realized for Amanda that any artifial jitter below 15 ns in the MC has no influence on reco accuracy or background rejection. We then said 10 ns just for safety. Then others came and said that for nearly vertical tracks close to a string scattering may be negligible and a 5 nsec request makes sense.” -- Christian NOT COVERED IN DETAIL IN THIS TALK

Our GPS clock ET-6000: Time to first fix: < two minutes Outputs operational: < five minutes Timing accuracy better than 2 µS & frequency accuracy better than 1E-8 Full system accuracy (100nS) within one hour. 10 MHz output, 1Hz output, Time burst output GPS from US Naval Observatory clock ~2x Accuracy

MCU Brick-Walls ET-6000 Specifications: 1Hz output is: positive (rising) edge on time, within ±100 nanoseconds relative to either UTC or GPS with six or more satellite averaging with 95% confidence. (± 150ns peak) 40ns RMS accuracy (jitter not specified) Cannot “Vote” multiple clocks; Neighboring clocks don’t track! x Tracking Algorithms in GPS clock “PLLs” x Variations in path, and multipath x Constellations and satellite switch-over

Unlock Behavior Power Up: + Sync to Satellites within an hour > Elapsed Time Format until “Tracking” - Hic-ups while searching; every 2 hours… Potential Loss of lock: x Clock Firmware (a-la TrueTime 2000 problem) x Power Outage x Misadventure (Murphy…) x Wind/Weather damage to Antenna (speculative) x National Security outage (speculative)

Flywheeling/Freewheeling TCXO Clock “continues” for hours Optional OCXO “continues” > 1 week Aging ±5 x per day, ±5 x per year Phase Noise Hz -94 Optional Rubidium Oscillator clock ~100ns per day slew WRT Aging < ± 5 x over 20 years, Phase Noise -90 ~ Hz Ru: Good short term stability, best hang time OCXO: best short term stability, good hang time Based on Symmetricom ET6000 series product specifications/experiences11

Implementation One GPS clock, an ET6000-OCXO Simple Passive fan-out (resistive splitters) All 9 DOM Hubs driven; All clock BNCs used! 0.35ns Jitter and Skew measured in situ at NPX DOR Firmware Improvements fixed NPX GPS “glitches” Not scalable to 90 HubsNot scalable to 90 Hubs

Implementation 2U chassis with 24 port fan-out (2 cards) o Modulates 1Hz signal o RJ-45 distribution cables carry 10 MHz, modulated 1Hz, & TVS o All balanced signaling < 0.6ns skew and jitter measured on the bench Passed MOAT: sps-ichub04, sps-ichub05, domhubjr, domhub51 Good Noise Immunity Stepping stone; scalable to needs of IceCube

Design Goals/Drivers > 90 ports plus Spares Meet accuracy, jitter and skew requirements Convenience: Single distribution cable per DOM Hub Measurability/Verifiability: Easy to confirm phase across all ports Reliability/Robustness: Quality components. No electrolytics. Noise Immunity: Balanced signals to DSB cards Minimize hard connections between racks: Magnetic coupling Modularity/Extensibility/Maintainability Hot-Swappable Port Cards Independent Port Drivers for each signal Low power Off the Shelf Components No heroic solutions Simplicity! (no programmable logic in clock distribution)

Signaling Details Balanced 10 MHz 500mV P-P through ‘ethernet magnetics’ - High common-mode immunity - Suppress EMI emission & RFI pick-up - Avoid ‘Ground Bounce’ pick-up in the counting house - Commodity components: compact, inexpensive… Modulate* the 1Hz signal (180deg Phase Modulation) * required to pass through ethernet magnetics… RS-422/RS-485 differential serial +12/-7V CM range Registered 1Hz U47-4 Modulated 1Hz 1Hz/1PPS 10 MHz

Rev 0 Fan-Out Card 12 Port 0.6ns port to port skew, worst pair <200 ps jitter Symmetry matters! Passes MOAT Revised DOR Firmware now supports Modulated 1Hz Inputs directly from GPS clock in Stand-Alone configuration Passes Fluorescent Lamp noise immunity test Status LEDs & Test Header

2006 Implementation and Beyond VME form factor conditioner card (1) in 2006 VME backplane fanout mezzanine card (1) in 2006 VME form factor 12 channel fan-out cards (9) proto now VME form factor Monitor card (1) 2006 goal (Least well defined) … DSB GPS Clock DOM Hub RJ-45 Cable (90+) Coax Cable (2) Serial Cable

Port Card Features, Rev 1 2-wideVME form factor; 12 RJ-45 port Common 10 MHz from Backplane Common modulated 1Hz from Backplane Common Serial from Backplane LVDS inputs, instead of direct GPS clock signals One Point Signal conditioning  2 minor schematic corrections  Skew tweaks  Additional Monitoring points: P2 and Header (e.g. demodulated 1Hz)

Backplane Fan-Out Active Piggy-back card mated to “Back-of-Crate” style VME backplane over Socket “2” Independently drive each EVEN socket Match phase at each driven socket 1-to-10 LVDS-to-LVDS fan-out chips Match skew to each driven socket Industry Standard “Back of Crate” style Backplane Piggy-back Card

Conditioner card Driven by GPS clock 10 MHz, 1Hz, and Serial Modulate 1Hz signal Control skew by design Ensure symmetry of port signals Drive LVDS backplane fan-out Status indicators/LEDs Prototyped on the Fan-Out card

Monitor Card Occupies an Odd Slot in VME card cage (e.g. #1) Implement GPS clock Monitoring specified by PDR Document Could contain FPGA, & SOPC, or SBC in a DIMM form factor Could report via ethernet Scope TBD

Watch List/Wish list “Lock” status from GPS clocks “Tracking” status from GPS clock “Power” status from GPS clock Parse time strings for error conditions Parse GPS Clock console port output Rich set of Status bits/words Satellite constellation Monitor phase offset between multiple GPS clocks

Verification Reference signals on header 10 pin header on DSB 10 pin header on Fan-Out card 4 pin header on DOR card (inside DOM Hub)

The Bottom Line Fan-Out card comfortably meets the 5ns Requirement Our GPS clock is a good choice, for the money No speed bumps this year On track for the final assault Single unit, flexible, modular design Builds on previous successes Avoids heroics and death marches. Fin

Verifications in situ IceCube AMANDA Opt 1: Measure Fiber Round-Trip time (2 fibers) Opt 2: Use Portable Atomic Clock - packaged PRS-10 - battery power Steps: 1. Measure/calibrate WRT IceCube clock 2. Transport clock to MAPO 3. Measure fiber distribution signals in MAPO against Atomic Clock 4. Transport clock back to IceCube Clock 5. Check Calibration Repeat 1 – to – 5 until satisfied Measure/Verify within 1ns should be achievable. 4”