Kostas Manolopoulos Tasos Belias University of Athens Physics Dpt. Electronics Lab. NCSR Demokritos CLBv2 Shore Station Kostas Manolopoulos Tasos Belias

Readout Electronics on Shore Shore station handles Data readout Control Calibration Main functions: Point to point data reception Broadcasting data distribution Clock/Commands distribution Time measurement of the optical network for each point  (calibration) GPS reference time stamping interface

Shore Station Overview WR Switch Server1 192.16.x.1 GPS PPS 10MHz Shore Sub Sea Switch Buffers DOM 1 2 16 12798 12799 12800 Server2 192.16.x.2 Server n 192.16.y.z Server Each DOM synchronizes to the absolute time (using White Rabbit) Each DOM receives a look up table with IP addresses while configuring the detector. All DOMs start at an absolute point in time which was communicated via a command over the White Rabbit network. All DOMs start their first time-slice at exactly the same time. All data is IP/UDP formatted and passed to the IP number corresponding to the time slice After ‘n’ time slices, first PC is again selected to process the data

Shore Station Functional Diagram DOM Buffer SFP Broadcast Optical Network Start Tx t4 Stop1 Reference Clock PTP Time Stamp t1 Time Stamp t4 t4 Stop2 t4 Stop3 t4 Stop4 Shore Station interface Rx j: DDMTD

Issues identified so far Adjust Switch Routing Table Software changes? PTP timestamps t1 same for all DOMs (broadcasted from the outgoing port) Firmware/Software changes MAC Control-Level multicast MAC addresses need to be handled correctly Broadcast of pause frames, control commands etc. will end up to every DOM Need to be able to address individual DOMs Implement address filtering?

More things to consider Inter-switch synchronization How to synchronize multiple WR switches… Required bandwidth to send data from the WR switches to their dedicated servers DAQ Networking: buffering and routing data of whole detector to dedicated processing units  10GbE, RDMA capability (?), traffic modeling to ensure scalability and robustness against burst rates Manpower 2 persons working in Athens currently (looking for more) Collaborate closely with 7Solutions Possibility to get IT-experts of departments involved ….

Next steps (outcome of this workshop) Separate tasks, prioritize and allocate them 7Solution collaboration can help speedup things Identify bottlenecks and critical paths U-Athens and NCSR Demokritos ready to undertake tasks Make a realistic time-schedule We want to expand our test-bench in Athens used to develop broadcast down-link & p2p up-link Kintex-7 available 2 WR switches needed 1GbE / 10GbE switch available

Lots to be done…