September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology KM3NeT CLBv2 1

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Visual Status 2 Rx_mac2buf I2C Fifo 31 TDCs TDC 0 Management & Control Data Control Wishbone bus RxPacket Buffer 64KB IP/UDP Packet Buffer Stream Selector (IPMUX) Rx_buf2data RxPort 1 RxPort 2 RxPort_m Management & Config. Tx_pkt2mac Tx_data2buf TxPort 1 TxPort 2 TxPort_m Flags Rx Stream Select TxPacket Buffer 32KB Flags Tx Stream Select 31 PMTs Pause Frame ADC Management & Control Hydrophone Fifo TDC 30 Fifo Nano Beacon GPIO Debug LEDs I2C Debug RS232 Temp Compass Tilt Point to Point interconnection Xilinx Kintex-7 Start Time Slice UTC & Offset counter since Time Slice Start MEM S 2 nd CPU LM32 M M WB Crossbar (1x7) WB Crossbar (3x2) S M S M M S S M M M SS S UART S M M S S M M State Machine SPI S M Flash UTC time & Clock (PPS+RefClk) aux_master ext_wb wrf_src wrf_snk

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Done: ◦ LM32 + WB-Crossbar + DPRAM + UART ◦ Soft-PLL FMC layout ◦ WR without PCI-express ◦ Deterministic PHY ◦ 1-wire Currently: ◦ Soft PLL (hardware + software). Still under study… ◦ Endpoint (= MAC) <= under investigation, focus on user interface ◦ Fabric redirector <= focus on user interface To do (in order of priority): ◦ Mini-nic <= Complex, but seems to work (PTP flows) ◦ PPS generator <= relatively straightforward ◦ SysCon <= easy? Status Listing 3

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology KC705 WR Ethernet port Block diagram 4 Ethernet traffic: PTP frames Ethernet MAC aux_master (Etehrbone) wrf_src wrf_snk

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Connect IPMUX => WRPC simulation 5 IPMUX Frame Generator Already reported last meeting (August 22, 2013): ◦ Initialize the Endpoint registers and memory ◦ Co-simulation of hardware & software (wrc.elf)

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Packet filter didn’t work properly ◦ No MAC address match because of one pipeline stage mismatch ◦ A newer version for “ep_packet_filter.vhd” was found in the repository that does the proper match! Endpoint investigation 1 ep_rx_path 6 fab_pipe ep_rx_early_address_matchep_packet_filterep_clock_alignment_fifoep_rx_oob_insertep_rx_crc_size_checkep_rx_vlan_unitep_rtu_header_extractep_rx_status_reg_insertep_rx_bufferep_rx_wb_master IPMUX xwrf_mux (= Fabric redirector) xwr_mini_nic 1 0

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Conclusion the White Rabbit PTP Core Release v2.0 tarball is a non coherent set of files! Many more files are updated. (A lot of bug fixes have been done since v2.0) Downloaded the newest files from the WR git repository, made minor KM3NeT changes and placed them into SVN Wrpc-sw Software also needs to be updated and setup for KC705 use => Mieke Bouwhuis Endpoint investigation 1 7

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology New set of WRPC files Packets received! 7 ping sent 7 received No replay since “WITH_ETHERBONE=0” Our own IP/UDP packet do not increment the RX counter since this packet does not flow thru the mini-nic!

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology CRC-errors occurred in the endpoint because the Ethernet frame generator (simulation) produces the wrong CRC (due to 8/16 bit CRC calculation). Endpoint investigation 2 ep_rx_path 9 fab_pipe ep_rx_early_address_matchep_packet_filterep_clock_alignment_fifoep_rx_oob_insertep_rx_crc_size_checkep_rx_vlan_unitep_rtu_header_extractep_rx_status_reg_insertep_rx_bufferep_rx_wb_master xwrf_mux (= Fabric redirector) IPMUX xwr_mini_nic 1 0

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Endpoint investigation 3; Bug? 10 Wrf_src_i: Should have been 0x0000!? Is this a bug in ep_rx_bypass_queue.vhd? => Needs to be sorted out! -- MAC Source and Destination Address and Ethernet Type-II FrameData.Frames(1, 0) := MAC_Dst(47 downto 32); FrameData.Frames(1, 1) := MAC_Dst(31 downto 16); FrameData.Frames(1, 2) := MAC_Dst(15 downto 0); FrameData.Frames(1, 3) := MAC_Src(47 downto 32); FrameData.Frames(1, 4) := MAC_Src(31 downto 16); FrameData.Frames(1, 5) := MAC_Src(15 downto 0); FrameData.Frames(1, 6) := x"0800"; -- IP Header FrameData.Frames(1, 7) := x"4500"; FrameData.Frames(1, 8) := x"0026"; FrameData.Frames(1, 9) := x"0000"; FrameData.Frames(1, 10) := x"0000"; FrameData.Frames(1, 11) := x"8011"; --check sum FrameData.Frames(1, 12) := x"B771"; --end check sum FrameData.Frames(1, 13) := x"C0A8"; FrameData.Frames(1, 14) := x"0102"; FrameData.Frames(1, 15) := x"C0A8"; FrameData.Frames(1, 16) := x"0103"; -- UDP Header FrameData.Frames(1, 17) := x"2000"; FrameData.Frames(1, 18) := x"1000"; FrameData.Frames(1, 19) := x"0012"; FrameData.Frames(1, 20) := x"0000"; -- Payload FrameData.Frames(1, 21) := x"0C00"; FrameData.Frames(1, 22) := x"1700"; FrameData.Frames(1, 23) := x"0000"; FrameData.Frames(1, 24) := x"0000"; FrameData.Frames(1, 25) := x"0000"; -- Padding FrameData.Frames(1, 26) := x"0000"; FrameData.Frames(1, 27) := x"0000"; FrameData.Frames(1, 28) := x"0000"; FrameData.Frames(1, 29) := x"0000"; fab_pipe ep_rx_crc_size_check 54 Length is okay Length is one short

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology ipmux needs more modification than expected… No rocket science involved but… … it takes time. Endpoint ipmux 11 ipmuxendpoint Connects to an 8-bit MAC= 16 bit MAC Adds MAC headerAlso adds MAC header depending on c_WRF_STATUS Fifo interfaceWishbone streaming interface

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Simulation now up and running. IP/UDP packets flow thru endpoint Need to check if last word of the packet lost? First shot (streaming wishbone b4) wrf_source -> ipmux_sink made... Thorough verification needed. Integrate clb_wrpc and ipmux. The design / simulation is getting big... General: endpoint, ipmux 12

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Timing Servo 13 No “TRACK_PHASE” yet No valid EEPROM image with calibration parameters in our case. Mieke Bouwhuis studies this issue

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Xilinx webcase answer from Erik Schidlack: ◦ “You cannot use the RXCDRLOCK output to check that the receiver is locked to data. This is only a coarse indicator and it is marked as reserved in the user guide too. Please check the incoming data stream to get this information. You should check for expected data.” ◦ “The behaviour you see now in simulation can happen in hardware too. It will be even more common. The signal will go low when the phase offset of the incoming data is greater than the current lock window of the CDR. This window is adapting to the signal and this situation can happen under normal circumstances when you still receive correct data.” My answer: ◦ “Checking for expected incomming data is only functional when you know what is to be expected, which is the case for idles but not for ordinairy data.” Workaround implement a digital filter, such that you only de-assert cdr lock when RXCDRLOCK gets de-asserted for at least "n" clock cycles. ◦ How big should "n" be? (currently set to “3”) PHY RXCDRLOCK_OUT 14

September 11-12, 2013KM3NeT, CLBv2 Workshop Valencia Peter Jansweijer Nikhef Amsterdam Electronics- Technology Focus on ipmux connection to WR ◦ Understand Endpoint and wr fabric sink and source => simulation Timing Servo (still) does not lock => PPS not enabled (but focus now on ipmux connection). ◦ Servo State: “Uninitialized” => “SYNC_SEC” => “SYNC_NSEC“ ◦ …but not yet “SYNC_PHASE” and “TRACK_PHASE” PHY RXCDRLOCK_OUT => workaround Summary WR status 15