May 29, 2013KM3NeT, CLBv2 Meeting Peter Jansweijer Mesfin Gebyehu Nikhef Amsterdam Electronics- Technology KM3NeT CLBv2 1

May 29, 2013KM3NeT, CLBv2 Meeting Peter Jansweijer Mesfin Gebyehu Nikhef Amsterdam Electronics- Technology Visual Status 2 Rx_mac2buf I2C Fifo 31 TDCs TDC 0 Management & Control Data Control Wishbone bus RxPacket Buffer 64KB IP/UDP Packet Buffer Stream Selector (IPMUX) Rx_buf2data RxPort 1 RxPort 2 RxPort_m Management & Config. Tx_pkt2mac Tx_data2buf TxPort 1 TxPort 2 TxPort_m Flags Rx Stream Select TxPacket Buffer 32KB Flags Tx Stream Select 31 PMTs UTC time & Clock (PPS, 125 MHz) Pause Frame ADC Management & Control Hydrophone Fifo TDC 30 Fifo Nano Beacon GPIO Debug LEDs I2C Debug RS232 Temp Compass Tilt Point to Point interconnection Xilinx Kintex-7 Start Time Slice UTC & Offset counter since Time Slice Start MEM S 2 nd CPU LM32 M M WB Crossbar (1x7) WB Crossbar (3x2) S M S M M S S M M M SS S UART S M M S S M M State Machine SPI S M Flash

May 29, 2013KM3NeT, CLBv2 Meeting Peter Jansweijer Mesfin Gebyehu Nikhef Amsterdam Electronics- Technology Done: ◦ LM32 + WB-Crossbar + DPRAM + UART ◦ Soft-PLL FMC layout ◦ WR without PCI-express Currently: ◦ Deterministic PHY => First shot White Rabbit in KC705 ◦ Soft PLL (hardware + software). First goal: lock onto a 125 MHz xtal and phase shift under control of LM32 via UART To do (in order of priority): ◦ Endpoint (= MAC) <= Complex! ◦ Mini-nic <= Complex! ◦ Fabric redirector <= probably less complex ◦ PPS generator <= relatively straightforward ◦ 1-wire, SysCon <= easy? Status Listing 3

May 29, 2013KM3NeT, CLBv2 Meeting Peter Jansweijer Mesfin Gebyehu Nikhef Amsterdam Electronics- Technology Integration 1 month from now… Ouch! Currently: ◦ Soft PLL (hardware + software) => 2 month? ◦ Endpoint (= MAC) 1 month? ◦ Mini-nic 1 month? ◦ Fabric redirector <= probably less complex ◦ PPS generator <= relatively straightforward ◦ 1-wire, SysCon <= easy? ◦ Connection Endpoint  IP-MUX => 2 weeks? Estimation: ~4,5 Month (if we are lucky)! ◦ Please also note: Peter is involved in another project the coming months! Planning (White Rabbit + IP-Mux) 4

May 29, 2013KM3NeT, CLBv2 Meeting Peter Jansweijer Mesfin Gebyehu Nikhef Amsterdam Electronics- Technology Integration of other objects ◦ White Rabbit + IP-Mux ◦ 2 nd LM32 system ◦ TDC’s / FIFOs / State Machine ◦ Hydrophone => Create complete design (hardware, software) + test environment (simulation) Planning Intergration 5

May 29, 2013KM3NeT, CLBv2 Meeting Peter Jansweijer Mesfin Gebyehu Nikhef Amsterdam Electronics- Technology GTXE2_CHANNEL IBUFDS_GTE2 gtx_dedicated_clk fpga_pll_ref_clk_101_p_i fpga_pll_ref_clk_101_n_i TXOUTCLK_OUT clk_gtx_i BUFG RXOUTCLK_OUT rx_rec_clk RXUSRCLK_IN RXUSRCLK2_IN TXUSRCLK_IN TXUSRCLK2_IN GTREFCLK0_IN rx_rbclk_o wr_gtx_phy_kintex7.vhd Entity: wr_gtx_phy_kintex7 Kc705_top.vhd Entity: kc705_top IBUFGDS clk_125m_pllref clk_125m_pllref_p_i clk_125m_pllref_n_i PLL_BASE Cmp_dmtd_clk_pll pllout_clk_dmtd Gc_extend_pulse ? Clk_i SPEC = FPGA_CLK_P/N Spartan6 pin G9/F10 SPEC = Spartan6 pin C11/D11 SoftPLL FMC DAC1 DAC2 VCXO 20MHz VCXO 25MHz CLK20_VCXO CDCM fpga_pll_ref_clk_123_p_i fpga_pll_ref_clk_123_n_i CPLLRESET_IN dac_dpll dac_hpll BUFG PLL_BASE Cmp_sys_clk_pll pllout_clk_sys Clk_20m_vcxo_i Timing reference (125 MHz) xwr_core.vhd Entity: xwr_core clk_ref_i(0) clk_fb_i(0) clk_dmtd_i clk_ref_i phy_ref_clk_i wr_core.vhd Entity: wr_core ????.vhd Entity: xwr_softpll_ng clk_ref_i clk_sys_i clk_dmtd_i xwr_endpoint.vhd Entity: xwr_endpoint clk_sys_i BUFG clk_dmtd_i phy_ref_clk_i HPC FMC only! Gtp_bitslide.vhd Entity: gtp_bitslide rst_done_n rst_i clk_ref, phy_ref_clk = TXOUTCLK (62.5 MHz) Clk_sys = used for synchronizing reset signals => must run always! (62.5 MHz) BUFG FPGA_CLK_P/N BUFG tx_out_clk_o clk_dmtd (62.5x MHz) 62.5 MHz PLL25DAC1_SYNC_N PLL25DAC_DIN PLL25DAC_SCLK PLL25DAC2_SYNC_N White Rabbit for Kintex7 Slave Clock distribution

May 29, 2013KM3NeT, CLBv2 Meeting Peter Jansweijer Mesfin Gebyehu Nikhef Amsterdam Electronics- Technology Tried first shot for White Rabbit on Kintex7 Solved issue with synchronous reset (reset switched off the clock ) Only found this through simulating the design!! KC705_top should do exactly the same as SPEC_top: First shot KC705_top 7 KC705_TOP hangs! Why?  Sherlock Holmes (Software/Hardware)

May 29, 2013KM3NeT, CLBv2 Meeting Peter Jansweijer Mesfin Gebyehu Nikhef Amsterdam Electronics- Technology 1. Switch Routing Table (software?) needs to be adjusted. 2. PTP timestamps t 1 for all DOMs are equal and reside in outgoing port (needs firmware- or software-change or both) 3. MAC Control-Level multicast MAC addresses; such as “pause frames” for flow control need to be handled correctly. Example: ◦ “DOM-B” request “Pause” = Okay (request over point to point link), but… ◦ “Shore station” request “Pause” may be problematic (request over broadcast link, all ports are stalled). Address single DOM? 4. Other surprises? Shore Station Broadcast brainstorm DOM A DOM B DOM C DOM D Buffer Port-2 SFP Broadcast Optical Network Start Tx t 4 Stop1 Reference Clock PTP Time Stamp t 1 Time Stamp t 4 t 4 Stop2 Time Stamp t 4 t 4 Stop3 Time Stamp t 4 t 4 Stop4 Time Stamp t 4 Shore Station interface 8 Rx  : DDMTD Rx  : DDMTD Rx  : DDMTD Rx  : DDMTD Port-1 SFP Main Electrical Optical Cable Port-3 SFP Port-4 SFP Port-5 SFP

May 29, 2013KM3NeT, CLBv2 Meeting Peter Jansweijer Mesfin Gebyehu Nikhef Amsterdam Electronics- Technology IEEE802.3 Clause 31 ◦ For example “Pause frames” for flow control (IEEE802.3 Annex 31B)  MAC Destination Address ◦ multicast C or unicast? (see Annex31B )  MAC Source Address  Length Type ◦ for “this is a MAC Control Frame”  MAC Control Opcode ◦ IEEE802.3 Annex31a => for “pause”  MAC Control Parameter ◦ Pause Quanta (1 Quanta = 512 bit times) Shore Station Broadcast MAC Control Level 9