WR & KM3NeT Peter Jansweijer

Artist impression & detection method KM3NeT Artist impression & detection method 100m ~ 860m 640 strings 18 DOM/string 11.520 DOMs 357.120 PMTs Volume: ~5 km3 Cherenkov m n 11.520 DOMs in the deep sea at 3-5 km depth

Digital Optical Module (DOM) Lower Hemisphere 19 PMTs Upper Hemisphere 12 PMTs Central Logic Board PMT Base: High Voltage Supply Analog Front-End

Central Logic Board (CLB) PMTs Lower hemisphere Xilinx KINTEX-7 (containing WRPC) HPC FMC Debug/Expansion PMTs Upper hemisphere SFP Readout 31 TDC’s with 1 ns bin size Other IO: Compass/Tilt, Acoustics, Nano beacon, Temperature White Rabbit enabled deterministic PHY for WR available: “wr_gtx_phy_kintex7.vhd”

Shore Station Demonstrator setup See presentation of 7-Solutions SPEC cards used for demonstrator. Later replaced by CLBs SPEC cards used for demonstrator. Later replaced by CLBs

Optical Network History: first idea Shore Subsea Digital Optical Modules Shore Station Optical network up to 100 Km, 80 wavelengths / fiber 4x18 DOMs Continuous Wave (CW) laser at the shore station. Reflective Electrical Absorption Modulator (REAM) in the sea. Pros: No lasers sub sea => reliable! tuning, temperature stabilization, repair, replacement at shore station Simple timing: Shore Station does time stamping (all electronics on shore) only need to add time delay “DOM to Shore Station” Easy measurement: Put the REAM in reflective mode, send a pulse back and forth and measure the transit time (OTDR) No complexity off shore => no need for Slow Control Sea cable is very expensive! => use many optical channels per fiber

1 Rayleigh Backscattering Bidirectional light transmission in the same fiber limited (~2 Km) due to Rayleigh Backscattering use two fibers for the long stretch Add extra path to “accurately” measure the time delay between DOM and shore station (OTDR method). After 100 Km, need a bit of amplification Shore Sub-sea < 1 Km up to 100 Km Only one DOM drawn for simplicity Idea: Pre calibrate time difference between forward and backward path…

2 Slow Control low extinction ratio Slow Control needed Solution: add “low extinction ration” modulation on the CW laser Slow Control Modulation (broadcast) CW Shore Sub-sea MOD Only one DOM drawn for simplicity

2a Slow Control Low Extinction Ration modulation was proven to work only in laboratory environment. More rigid to use a separate Slow Control wavelength! Shore Sub-sea Slow Control (broadcast) Only one DOM drawn for simplicity

3 Single pulse delay measurement? Measuring time with a single pulse is not so accurate… Solution: White Rabbit “Every 125 MHz tick is put to good use” [1] Shore Sub-sea Slow Control (broadcast) Only one DOM drawn for simplicity The idea: Network asymmetry can be modelled in a revised “WR Link Model” [1] The White Rabbit Project https://espace.cern.ch/be-dep/CO/ICALEPCS%202009/1158%20%20The%20White%20Rabbit%20Project/TUC004_FINAL.pdf

4 REAM to SFP The REAM is expensive and single supplier => SFP Slow Control (broadcast) Shore Sub-sea

Are we there? Need a dedicated WR implementation for the Shore Station See presentation this afternoon by 7-Solutions Non-standard Ethernet: No Auto Negotiation, packet routing, pause frame handling Calibration is not simple… The idea to model network asymmetry in the “WR Link Model” needs knowledge of fiber length of either the broadcast channel, or the individual DOM-> Shore Station links. SFP = laser sub-sea We may loose the DOM-> Shore Station communication due to wavelength shift over time. Need possibility for laser tuning over a unidirectional communication channel. Diverting from WR main development Need to apply KM3NeT adjustments to new firmware/software releases. Connecting to standard Ethernet equipment (for tests) needs constant attention.

Timing Calibration Using WR as a ps accurate measurement tool SoftPLL t(l) D(l) KC705 Reference: IEC 86A/1419/CDV OPTICAL FIBRES Chromatic dispersion measurement methods Usual WR calibration => Measure PPS differences and tweak a KM3NeT challenge => Measure a and put it into the system and rely on accurate PPS alignment (we cannot probe it!)

Using WR as a ps accurate measurement tool Measure RRT for 8 different wavelengths and calculate propagation delay of fiber for 8 wavelengths as all other delays are measured as a reference on forehand and known. Loop back at the far end fiber in under-sea cable (length ~95 km) EDFA (for MEOC Italy) f2 SFP f1 SFP Att. 15dB f3 SFP in WR Master: transmitter: 0 till 5dBm receiver: -24dBm (pindiode) SFP in WR Slave: transmitter: 2 till 5dBm receiver: -35dBm (APD)

Using WR as measurement tool courtesy to Mar v. d Using WR as measurement tool courtesy to Mar v.d. Hoek, Gerard Kieft and Antonio d’Amico, Nikhef Amsterdam Gerard Sellmeier equation: “an empirical relationship between refractive index and wavelength” Group delay t(l) Antonio

WR synchronization after restart courtesy to Diego Real, David Calvo IFIC Valencia and 7-Solutions Once in a while “wrong” synchronization (11/1200) Version 3.3 switch software (test with V4 ongoing) SPEC f_eval_sync_detect_threshold = 8192 CLB f_eval_sync_detect_threshold = 4000000

Future? (KM3NeXT)? 1 fiber = 80 wavelengths = 40 bidirectional 1Gbps links = 40 WR switches ~ 9 x 40 DOMs = 360 DOMs (note: 4 x 18 = 72 DOMs per fiber in the current topology) 1x18 DOMs (Or 2x9 @ ~100 Mbps/DOM) Shore Sub-sea Actual rates based on pessimistic PPM-DU data (10 KHz / PMT) (see slide 9 of Marco Circella, Steering Committee July 29, 20154 https://indico.cern.ch/event/328787/session/1/contribution/8/material/slides/0.pdf) 10 KHz rate * 31 PMTs / DOM = 310 KHz / DOM 6 bytes / event => 6 * 310 KHz = 1,81 MB/s => ~ 20 Mbps / DOM

KM3NeT Future? (KM3NeXT)? Pros: Calibration issues solved Standard WR (keep up with software/firmware updates) Standard Ethernet: Standard Pause frame Flow control No complicated packet routing Auto negotiation plug and play with industry network cards => easy testing. Optical network with industry standard DWDM may be more relaxed when data is accumulated. No broadcast link bandwidth limit or single point failure Cons: Need development for a 15 year reliable switch in a new form factor. Need subsea switch maintenance. General remark: A wish for 10 Gbps (data) uplink ports at first level Shore Station switches. KM3NeT

Thank you

Backup Slides

Stream Selector (IPMUX) CLB block diagram IP/UDP Packet Buffer Stream Selector (IPMUX) 31 TDCs Start Time Slice UTC & Offset counter since Fifo TDC0 Time Slice Start RxPacket Buffer 64KB RxPort 1 31 PMTs RxPort 2 Fifo TDC 30 Rx_mac2buf Rx_mac2buf Rx_buf2data Rx Stream Select Flags RxPort_m Management & Control aux_master S 6 1 7 Pause Frame Management & Config. State Machine 5 ADC Fifo 4 Hydrophone 2 3 TxPacket Buffer 32KB TxPort 1 Management & Control WB Crossbar (3x3) S0 M1 S1 S2 M0 M2 TxPort 2 S Tx_pkt2mac Tx_pkt2mac Tx_data2buf Tx Stream Select Multiboot Flags TxPort_m S Management & Control S WB Crossbar (1x8) M M M S M Debug LEDs M M M M M M S GPIO 2nd CPU LM32 M S S S Xilinx Kintex-7 MEM S WB Crossbar (3x3) S0 M2 S1 M0 M1 S2 SPI UART I2C3 I2C1 S I2C2 S Data UTC time & Clock (PPS, 125 MHz) Control Point to Point interconnection SPI Flash Debug RS232 Temp Tilt & Compass Nano Beacon Wishbone bus

Optical network