September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Shore station brainstorm 1

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology 1. Switch Routing Table (software?) needs to be adjusted. 2. PTP timestamps t 1 for all DOMs are equal and reside in outgoing port (needs firmware- or software-change or both) 3. MAC Control-Level multicast MAC addresses; such as “pause frames” for flow control need to be handled correctly. Example: ◦ “DOM-B” request “Pause” = Okay (request over point to point link), but… ◦ “Shore station” request “Pause” may be problematic (request over broadcast link, all ports are stalled). Address single DOM? 4. Other surprises? Shore Station Broadcast brainstorm DOM A DOM B DOM C DOM D Buffer Port-2 SFP Broadcast Optical Network Start Tx t 4 Stop1 Reference Clock PTP Time Stamp t 1 Time Stamp t 4 t 4 Stop2 Time Stamp t 4 t 4 Stop3 Time Stamp t 4 t 4 Stop4 Time Stamp t 4 Shore Station interface 2 Rx  : DDMTD Rx  : DDMTD Rx  : DDMTD Rx  : DDMTD Port-1 SFP Main Electrical Optical Cable Port-3 SFP Port-4 SFP Port-5 SFP

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Other surprises: PTP t 1,t 4 timestamps over multiple switches 3 PTP t 1 Broadcast PTP t 4 DOM inputs PTP t 4 DOM inputs PTP t 4 DOM inputs 1. Redistribution of PTP t 1 not only within one switch… 2. Routing PTP frames is an issue (layer-2; no IPv4!)… Note that t 4 needs to be routed to the appropriate DOM via the broadcast channel. …over multiple switches (depending on detector broadcast network segmentation) This needs brainpower! One t 1 Many t 4

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Brainstorm-1 4 PTP t 1 Broadcast PTP t 4 DOM inputs PTP t 4 DOM inputs PTP t 4 DOM inputs 1. When layer-2 is not routable then distribute the timestamps using your custom protocol. 2. … kind of awkward! And it may consume switch ports 1 Gbps WR 10Gbps COTS 1 Gbps WR

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Brainstorm-2 5 PTP t 1 Broadcast PTP t 4 DOM inputs PTP t 4 DOM inputs PTP t 4 DOM inputs 1. Layer-2 is not routable, then… 2. Wrap PTP t 1 in a layer-3 message and route this to the rest of the switches. 3. Each timing input calculates timing servo action and passes this to the broadcasts port. 1 Gbps WR 10Gbps COTS 1 Gbps WR

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Tempting thoughts (that shouldn’t be thought at this stage in the design cycle…) 6

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Current major troubles in the design 7 Optical network with GHz is very tempting… (will it be possible?) Broadcast networking is difficult to implement. Broadcast network increases “asymmetry” =>this complicates timing calibration. Each DOM has a different wavelength laser. This is not disastrous but requires more careful tracking of DOMs.

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Tempting idea: a WR-Switch in the DU (Paul Kooijman) 8 1 Gbps WR Shore 1 Gbps WR 1 Gbps WR, 10Gbps data Sub-Sea A tempting idea to put a WR switch in each DU! But it solves lots of problems!

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Tempting idea: a WR-Switch in the DU (Paul Kooijman) 9 One WR switch per DU (1x16 or 2x8 DOMs, later redesign 1x18 or 2x9 DOMs). + Better bandwidth utilization per wavelength => + Less wavelengths needed => possibility for bidirectional network + Even with bidirectional network the optical network easier! + cost effective! Inexpensive SFPs in the string and a limited amount of expensive long haul SFPs (no needed EDFA’s anymore) + No broadcast anymore! All point to point. Timing is distributed as it is meant to be => “Normal network” + All DOMs can be the same (colorless) - More complexity sub-sea. - Shore station switch must handle 10 Gbps (WR only available on 1 Gbps) - SFP reliability not proven! 6 x plus! Only 2 minus 1 general worry!

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Tempting idea: distributed WR-Switch(es) in the CLB (Henk Peek) 10

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology 11 + Redundancy + Better bandwidth utilization per wavelength => + Less wavelengths needed => possibility for bidirectional network + Even with bidirectional network the optical network easier! + cost effective! Inexpensive SFPs in the string and a limited amount of expensive long haul SFPs (no needed EDFA’s anymore) + No broadcast anymore! All point to point. Timing is distributed as it is meant to be => “Normal network” + All DOMs can be the same (colorless) - More complexity sub-sea. - Needs further CLB development (implement simple static switch functionality) - Shore station switch must handle 10 Gbps (WR only available on 1 Gbps) - SFP reliability not proven! 7 x plus! 3 x minus 1 general worry! Tempting idea: distributed WR- Switch(es) in the CLB (Henk Peek)

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Conclusion on tempting thoughts 12 These thoughts shouldn’t be thought at this stage in the design cycle… But please let us re-think before proceeding! We are designing an awkward system right now. If we take a step back now we may have an operational KM3NeT detector faster! I know… This involves all disciplines: ◦ Hardware ◦ Software (+ Networking) ◦ Mechanics ◦ Firmware ◦ Optical network ◦ Power

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Backup slides 13

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology 7500 Hz hitrate/ PMT 48 bit/hit => 7500 * 48= bps/PMT 31 PMTs / DOM * 31 ~ 12 Mbps / DOM 18 DOMs / string 12 Mbps * 18 = 216 Mbps / string = 21% of a 1 Gbps channel Bandwidth 14

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology WR I2C buses 15 Soft PLL FMC SFP EPROM SCL SDA SFP+MOD_DEF1 SFP_MOD_DEF2 C10 C18 SCL SDA SCL SDA C30 C31 KC705 FMC-J22 FPGA H30 B28 K21 L21 U49 PCA9548 (I2C mux) IIC_SDA_MAIN IIC_SCL_MAIN clb_xwr_core xwc_periph SCL SDA SFP_SCL SFP_SDA SYSCON Regs GPIO Set/Readback Register Bit 2: SCL Bit 3: SDA Bit 8: SFP_SCL Bit 9: SFP_SDA Ch 1