Timing and data-exchange between CERN and LNGS The CERN and LNGS UTC systems The beam data exchange database The early warning signal D.Autiero/IN2P3 Lyon, 6/4/06

The Global Positioning System (GPS) The GPS system was built and it is controlled by the US militaries (DOD), it is composed by at least 24 satellites orbiting at Km Depending on the place and the time of the day on average between 5 and 8 satellites are visible by an observer on the earth

Each satellite is equipped with a Cs atomic clock and transmits its time and position (computed with its ephemeris): e.g.: x1,y1,z1,t1 The observer on the earth, getting the quadrivectors of 4 satellites has to solve a system of 4 equations in order to find his own time and position: x0,y0,z0,t0 (x0-x1) 2 + (y0-y1) 2 + (z0-z1) 2 = c 2 (t0-t1) 2 (x0-x2) 2 + (y0-y2) 2 + (z0-z2) 2 = c 2 (t0-t2) 2 (x0-x3) 2 + (y0-y3) 2 + (z0-z3) 2 = c 2 (t0-t3) 2 (x0-x4) 2 + (y0-y4) 2 + (z0-z4) 2 = c 2 (t0-t4) 2 Then x0,y0,z0,t0 can be converted in the more familiar Latitude,Longitude,Altitude and time

GPS UTC clock system. Built in 1991 by the company ESAT located in Torino: Macro detector paper: NIM A 486 (2002) INFN/TC-01/2001

Oscillator adjustment: Each second the local clock scale is compared to the GPS (20 ns units). The average of 60 of these time differences is produced every minute (  t) 44 ns RMS Master clock unit = GPS unit + 10 MHz Rubidium oscillator The GPS works normally in time only mode (its position is already very well known) Rubidium oscillator: Short term stability (Allan variance) better than 3x (  ) over 100 s Long term stability (aging/temperature) about 1 micros/day Long term variations are adjusted using a linear fit of 180  t points (every 3 hours). After adjustment the residuals of the local time to the GPS are at the level of tipically 15 ns RMS on  t

LNGS time distribution system to underground labs: 2 independent master clock units are installed in the external laboratory. The UTC time scale is known at better than 100 ns. Every ms (1000 better than the 1 PPS standard) a synchronization pulse and the time/date string are sent through the 8 Km long monomodal optical fiber (1310 nm) to slave clocks in the caverns. The ligth pulse is converted in electric serial pulses. GPS antenna GPS receiver Master clock Slave clock(s) External LAB Underground LABs Rb clock Propagation time is measured once and subracted systematically at the syncronization level 8 Km mono-modal 1310 nm optical fiber Optical signal sent every ms Edge (start of the ms) 80 bits code Date etc …

Slave clocks have to regenerate the local time with 10 MHz TCXO oscillators with a stability of 1 PPM (it has to keep on track over 1 ms). The time is then available in a 80 bits format and it is known at better than 100 ns MACRO was reading it with CAMAC modules BOREXINO with VME OPERA built its own slave clock unit (with much improved performance) completely compatible with the existing optical fiber distribution sistem

The OPERA master clock card PCI cards replacing the old slave clock and providing the clock system in OPERA Totally compatible with the LNGS clock system (ESAT consultancy) Developed at IPN Lyon in 2003 Operative at LNGS since beginning of 2005 PPmS from optical fiber 10 MHz ovenized quartz local oscillator (reset every ms) almost a Rb clock: Allan variance 2* over 1s T stability 5* °-50° Aging 7*10 -8 over 5 years Master clock signal edge tracing within 20 ns

The OPERA clock distribution system Master clock O/E converter Master card 0 Node card i SM2 SM1 1:2 splitter PCI card Optical fiber MLVDS 8 Km mono-modal 1310 nm optical fiber Sync. signal sent every ms (PPmS) Time stamp distributed in units of 100 ns to the detector nodes 10 MHz high stability local oscillator The FE electronics works in units of 10 ns. The DAQ works in cycles of 1 s (up to 256) with a fine counter of 10 ns. Delays are measured and subtracted at the nodes levels. The DAQ counters are cross-referenced with the UTC date stored by the master card

CERN system Symmetricom (ex True-Time) XL-DC system Oscillator Temp. Stability Standard receiver specifications 150 ns peak to peak Rubidium oscillator Also used in K2K and MINOS

The LNGS system in the past was quoted with an overall UTC tracing accuracy better than 100 ns We are aimining for CNGS at having two systems with comparable accuracies (<100 ns) which have then to be combined togheter. We planned some cross-checks: 1)The CERN system was sent to the Swiss metrology institute METAS. They performed a calibration and released a certification 2) We organized to cross calibrate the two systems by bringing the CERN clock to LNGS and cross tagging the two systems for a long period. (this will not include all what comes after the master clock) + remeasurement of the optical fiber delay Which accuracy ? XL-DC standard also used in K2K and MINOS with the goal of « 100 ns » accuracy, what does it mean ? K2K XL-DC:

CERN-LNGS UTC clocks intercalibration Performed in collaboration among: J.Serrano. P.Alvarez, J.Lewis (CERN), D.Autiero (OPERA), S.Parlati, G.Di Carlo(LNGS) : Put the two UTC systems side by side: CERN system brougth to LNGS on 7/3/2006 and reinstalled with its own antenna and a DAQ chain and a time interval counter (300 ps res.) in order to monitor the time difference of the output of the two systems as a function of time. Data taking performed under various conditions till yesterday (5/4) The CERN system was calibrated vs UTC a few month ago by the Swiss metrology institute METAS (including antenna and 30m of cable) The cable delay is correctly accounted in the configuration Both CERN and LNGS will have a system with double units

Clock 1 active (backup) LNGS: Clock 2 active and connected to underground lab Clock 1 on, not connected (backup) 10 MHz Rb oscillators Clock 3 (off) Optical fiber transmitters PC for monitoring and control Clock 1 Clock 2 Active, connected to underground

CERN system Symmetricom (ex True-Time) XL-DC system (also with rubidium oscillator) CERN system installed in the LNGS UTC clock room with the DAQ chain PC+labview Fluke time interval counter PM 6681 GPIB-ethernet interface Digital scope

Long run in auto-> time mode 10/3-22/3 (> 1 million of measurements, 1/s) 10/3 22/3 Time difference stabilizes on a plateau around 350 ns (clock2 anticipates CERN) Converged after 2 days to time mode with: N42° 25’ 15.4’’ E13° 30’ 53.0’’ Alt 1035 m asl Time (s) x 10 3 CERN-clock2 (ns)

1 day=86400 s Plateau 355 Band of +-23 ns 1 day Good cross-tracking of the two systems, which is the origin of the offset ? CERN-clock2 (ns) Time (s) x 10 3

Fine details of the time difference evolution

LNGS UTC Clocks block diagram nanostepper

Discovered that the two LNGS UTC clocks which should be identical (clock2 and clock1) have an offset about of 140 ns, quite stable with time. Different software configuration ? How the antenna cable delays are taken into account in the configuration of the two systems Measurement 22/3 Measurement 8/3 ns

Clock1-clock2 monitored over 13 hours

The coordinates of the antenna used by clock2 are known from precise measurements: N42° 25’ 15.6’’ E13° 30’ 52.8’’ Alt 985 m asl WGS84 The CERN system if off by 50 m in altitude, it comes out that the same bias was noticed also at CERN (50m=167 ns). Both at CERN and LNGS this is confirmed also by other portable GPS systems Apparently both systems refer to the geoid model WGS84. Difference under investigation Perform the same measurement with the CERN system in time only mode with forced LNGS coordinates The CERN system converged to the following coordinates: N42° 25’ 15.4’’ E13° 30’ 52.9’’ Alt 1037 m asl WGS84

WGS84

Long run in time mode with CERN system forced to known LNGS coordinates 22/3-5/4 5/4 22/3 Time difference stabilizes on a plateau around 240 ns (-110 ns) Time (s) x 10 3 CERN-clock2 (ns)

ns CERN, Auto coordinates CERN, Forced LNGS coordinates =355.6 ns =248.5 ns

Complete calibration of the time distribution chain: Telecom room Telecom room

Conclusions for the UTC intercalibration : 1) The two GS clocks differ by 140 ns, check with ESAT their software configuration and how the antenna cable delay is accounted for the two 2) The CERN system tends not to converge to the correct coordinates by overestimating altitude by 50 m (a cheap GPS does it fine). Checks with METAS in progress, WGS84 ? 3) Once reached the plateau the two systems (XL-DC and clock2) are tracking each other in a band of 40 ns (+-20) and an offset of 356/248 ns. This is a quite nice result provided the offsets that we found are completely understood and under control 4) A viable scheme has been setup in order to complete the calibration of the LNGS time distribution system by measuring the delays (+jitter) of the chain The CERN system today has been dismounted and sent back to CERN (it is needed for the machine startup) First it will be sent to METAS with all the new informations for a cross-check, contacts with ESAT as well. We aim at fully understand the offsets Work in progress

The CNGS-LNGS database Needs: 1)We need to correlate the events recorded by the experiments with the beam spills windows (10.5 microseconds). This is done through the UTC times recorded at CERN and LNGS: T(LNGS)=T(kicker)+TOF, accuracy better than 100 ns. We need the list of spills UTC times from the CERN database 2)We need also to have online some rough, synthetic spills quality informations (intensity, status word) 3)We need a full database of the beam conditions for physics analysis and comparison with the beam simulations, (e.g. NOMAD) important for numu->nue analysis 4)We need to record in the beam database the feedback of the far (LNGS) beam monitoring: muons from neutrino interactions in the rock, neutrino interactions in the detectors 5)We need to be able to consult at LNGS the beam status, like it was done at WANF with a graphical application

The CNGS database (ORACLE based) produces 1 TB/day of information, mostly technical and it is in the hidden accelerator controls network at CERN (LHC logging database) After a few meetings among OPERA/CNGS and the CERN Database people it was decided to implement a new gataway server (ORACLE) on the public network in order to exchange the relevant informations among CERN and LNGS for the data-taking, beam monitoring and beam analysis: The data will be available on the public gateway about 15 minutes after recording in the main database The database is structured per CNGS cycles The server will be bidirectional in order that also LNGS experiments will be able to write the events observed in correlation to the spills (some discussions started in OPERA and LVD) A new server has been bougth A selection of a subset of the variables of the main database to be copied for a complete beam physics analysis has been discussed among Edda,Paola and myself

Status: (Ronny Billen) Scheme under test in a development environment (not yet with the final server machine) Database scheme ready Data transfer procedures under development Definition of CNGS relevant variables (99 up to now) ongoing Data flow will be tested as soon as the DAQ chain allows for it The user documentation will be provided and made available on the CNGS site

Early warning signal It is useful to know in advance the arrival time of neutrinos in order to prepare the DAQ and prevent operations which could interfere with the neutrino events recording (e.g. OPERA from time to time performs calibrations with flashing LEDs) The next neutrino spill time can be predicted in advance of several seconds. The information can be transmitted through the network Two fast extractions/cycle 10.5 micro seconds Separated by 50 ms EW Window for calibrations

SPS Cycle Proposal for 2006 –Commissioning: 12s FT + 6s CNGS –Run 1 12s FT + 6s CNGS + 4.8s MD –Run 2 12s FT + 3x6s CNGS + 4.8s MD

An experiment like OPERA is continously taking data, independently of the beam informations (data-base and early warning access through the network): 1) The beam informations are needed to validate the bricks extraction once per day 2) The calibrations can be conservatively skipped for some time in case the early warning signal does not arrive. In case the EW is there the calibrations will be performed till a conservative window around the neutrino spills The transmission of the early warning from CERN to LNGS has been implemented and tested since the beginning of March by J.Lewis on the OPERA DAQ gateway computer 1) It is based on UDP packets. The transmission time is around 10 ms 2) The packets are sent every 1.2 s (all possible SPS cycles are multiple of 1.2s which is the PS-booster cycle). The packets contain the date of the future neutrino extraction. This date can be the same for many packets of the same cycle. The idea of the 1.2 s repetition is to keep a constant rate link among CERN and LNGS 3) The DAQ looks at the arrival time of the packet, compares to the extraction time written in the packet and decides if there is time or not to perform calibrations

The UDP packets can be transmitted from the CERN machine also to other LNGS nodes (we have to decide if it is more convenient to dispatch from CERN to many LNGS nodes or to use one LNGS node as repeater) The documentation will be put on the CNGS site