System On Chip Offshore Node S. Anvar, H. Le Provost, Y.Moudden, F. Louis, B.Vallage, E.Zonca – CEA Saclay IRFU – Amsterdam/NIKHEF, 2010 July 5

RTOS DDR2 Memory Flash Memory Processor boot The proposed KM3Net Off-Shore Processor Board Processor Slow Control Slow Control Task Slow-Control (SC) for the Storey 1Gb/s Ethernet Link To shore station Data Task Data Digitized data Readout Logic Readout System On Chip (RSOC): One Component SC Protocol Logic Clock Extraction SCOTT Front End ASICs

The RSOC demonstrator (Antares upgrade) PPC405 (Virtex-4) processor board Analog Ring Sampler (ARS Asics) Data Gigabit Ethernet Antares Run Control & On-Shore Data Acquisition Off Shore Upgrade is Fully Functional : PPC405 is running vxWorks Software recompiled for PPC405 Firmware adapted to IPIF Xilinx Interface

RSOC Ethernet throughput XILINX ML507 PC DELL OPTIPLEX 760 ETHERNET SWITCH HP ProCurve G Ethernet 10/100/1000 over Copper NFS Server TFTP Server PPC440 running vxWorks OS at 400MHz and a PLB bus frequency at 100 MHz TCP/IP Ethernet transmit throughput of about 988 Mbit/s. It is coherent with the results given in xapp1063 Note : XILINX ML507 Ethernet Throughput, Bendahmane Rafik, Le Provost Hervé, , CEA-Irfu

KOALA Board (KM3Net Optical module Acquisition board for LocAl Processing) PRISM VIRTEX4-FX Processor module Serial Copper Ethernet gigabit link Scott OM 0 Scott OM 2 ScottOM 4 Instrumentation Compass NanoBeacon Humidity (I2C) Temperature (SPI) OM 1 OM 3 OM 5 HV PMT Switch 48V (I2C) DAC & ADC (SPI) Serial

Pmt Adaptation Switch 48V Hight Voltage Control PMT Amplifier Power supplies PRISM200 Temperature Controller Connectors for: Humidity Controller (1) Piezzo Compass Chip (2) Nanobeacon PRISM200 Optical link SFP ASIC SCOTT 1 2 KOALA Board Layout

Use the recovered carrier frequency of the gigabit serial Ethernet link. Preserve the standard Ethernet protocol. Experimental setup : two ML507 Virtex 5 evaluation boards - Requires fixed or known latency and the ability to assign round trip latency to each of both one way transmissions. Clock distribution* : Principles and requirements *The following slides were prepared by Yassir Moudden It gives a status of his work on the clock and commands distribution

bit_slide Custom comma alignment logic GTX Embedded hardware TEMAC Custom Ethernet frame generator Simple client (address swap ) TXRX TX RX 8 bits is_aligned bit_slide [ ] REC_clk1 REC_clk2 TX_clk TXPMASETPHASE = ‘1’ TXENPMAPHASEALIGN = ‘1’ NB : 16 bit idles are synchronized with 62,5 MHz TX_clk NB : TX and RX elastic buffers bypassed Clock distribution : Design Basics

Comma Detection - Word alignment : Counting bit slides to recover REC_Clk1 phase Skew ( TX_clk, REC_clk1) in ns Number of bitslides to achieve comma alignment after full system RESET - Because of serialisation / deserialisation of 20 bit words, REC_clk1 is skewed with respect to the propagated TX_clk. - GTX can compensate skew to realign received data with REC_clk1. - Custom realignment required to track number of bitslides and latency compensation in multiples of 0.8 ns.

Histogram range ~ 90 ps standard deviation ~ 15 ps mean = ns Reconstructed clock Rec_Clk1

Comma Detection – Word alignment --Comma Detection and Alignment Attributes ALIGN_COMMA_WORD_0 => 2, COMMA_10B_ENABLE_0 => " ", COMMA_DOUBLE_0 => FALSE, DEC_MCOMMA_DETECT_0 => FALSE, DEC_PCOMMA_DETECT_0 => TRUE, DEC_VALID_COMMA_ONLY_0 => TRUE, MCOMMA_10B_VALUE_0 => " ", MCOMMA_DETECT_0 => FALSE, PCOMMA_10B_VALUE_0 => " ", PCOMMA_DETECT_0 => TRUE, RX_SLIDE_MODE_0 => "PCS ", RXDATAWIDTH1 => "00", TXDATAWIDTH0 => "00", INTDATAWIDTH => '1', PROBLEM : TEMAC input / output data bus on GTX side is 8 bits wide. bit_slide An invalid configuration of the GTX enables 20 bit alignment on receiver side provided 16 bit idles are aligned with the 62.5 MHz TX_clk. GTX alignment attributes Xilinx GTX documentation Custom comma alignment logic GT X Embedded hardware TEMAC TEMAC client RX TX 8 bits is_aligned [ ] 16 bits CUSTOM LOGIC TO ALIGN 16 BIT IDLES WITH 62.5 MHz TX_CLK Currently investigating the possibility of inserting logic between TEMAC and GTX to have GTX operate with 16 bit data and to keep 16 bit idles aligned with 62.5 MHz clock.

Synchronous Commands Outgoing commands : Need for synchronous commands sent to the remote stations without disturbing the Ethernet flow. In simulation, the above simple design (GTX, TEMAC and inserted logic) proves able to handle the necessary low rate of synchronous commands. - command to be inserted is stored in long path pipeline - switch to long path : move command out and pipeline Ethernet flow - switch back to default short path within inter frame gap by removing one 16 bit idle. GTX Embedde d hardware TEMAC TEMAC client RX TX 8 bits RX_clk Select RX short or long path RX short path RX long path (DEFAULT) TX_clk Select TX short or long path TX short path (DEFAULT) TX long path - commands detected inside RX default long path pipeline. Could use a specific K-char to tag commands and second byte to define 256 commands. - switch to short path, remove command from RX pipeline and replace it with 16 bit idle. - switch back to default long path within inter frame gap, which is then one idle longer than initial gap. Incoming commands : N.B. The data bus is pipelined along with several other (running disparity, char_is_K, etc.)

Transmission from offshore to shore using REC_clk1 REC_clk1 is clean enough at least for lab work. Total number of bitslides to achieve comma alignment offshore and back on shore after full system RESET Skew ( TX_clk, REC_clk2) on a round trip in ns However, our configuration of the PMA phase alignment induces some unexpected effects that need to be understood. An additional random 1.6 ns is included in the measured REC_clk2 / TX_clk round trip skew after system reset. ( N.B. The alignment of 16 bit idles out from the TEMAC is random after system reset. )