The White Rabbit MCH Javier Serrano on behalf of Tomasz Włostowski BE-CO-HT 12 October 2010 Talk Plan (probably too ambitious for 10 minutes) These are not the notes for this slide but rather the plan for the “ideal talk” on the WR MCH, which would last roughly 20 minutes. Intro and context This is the work of Tomasz Wlostowski, fellow in BE-CO-HT. The White Rabbit (WR) Project: 1000 nodes, 1 ns, 10 km fibers. The WR switch. Clock recovery, PTP, etc. Why an MCH: initial ideas, other uses (IN2P3, Elettra). MCH design. Overall architecture. UTCA usage: GbE, UTC time code, corrected clocks, xTCA 4 Physics compliance... Timing FPGA. Main FPGA. Watchdog CPU. I2C hub. Main CPU. Boot and development environment. Conclusion and outlook. uTCA right platform for this development. HW debugged, gateware and software underway. MCH v3 (probably Virtex-6 based) in technical spec phase. See more at http://www.ohwr.org/projects/white-rabbit

Introduction and context The design Conclusion and outlook Quick slide.

The White Rabbit Project Introduction and context The White Rabbit Project An Ethernet-based network for synchronization and determinism. 1000 nodes, 1 ns precision with typical fiber lengths of 10 km. Compliant with existing standards (e.g. IEEE1588 aka PTP). White Rabbit is a multi-laboratory project to design a deterministic network with good (sub-ns) synchronization capabilities using existing standards as far as possible (Ethernet, PTP, etc.) . A typical network could have 1000 nodes with typical fiber lengths of 10 km. More information in the reference at the end of the talk.

The White Rabbit Switch Introduction and context The White Rabbit Switch The WR switch is a key ingredient of the WR network, and it's currently made of an MCH card and a mini-backplane. A switch has two uplinks (for redundancy) and (currently) 8 downlinks. The clock gets recovered from the uplink data stream and used to encode data in the downlink ports. Through this mechanism, plus delay compensation using enhanced PTP, we can have a common distributed clock everywhere in the network.

Why an MCH Introduction and context The initial idea was to build the switch completely as a uTCA system. However, this was too expensive and we are currently favoring a mini-backplane-based approach. Even with this new strategy, an MCH is still an attractive format for two reasons: - Some people (IN2P3, Elettra...) might be interested in distributed synchronization of uTCA-based data acquisition systems, i.e. populating the AMC slots with ADC cards which receive the clocks from the backplane. - We ourselves will have to deal with a legacy timing system, and plan to use converters from new to old timing format in AMC form factor.

The sandwich MCH Design The WR MCH is a full-height single-width AMC board. It is made of a sandwitch of 4 PCBs: - PCB1 contains the main (ARM9) CPU, the watchdog (ARM7) CPU, the main FPGA, and memories among others. On the front panel it exposes the ARM9 100 Mb/s Ethernet interface through an RJ45 connector. Towards the backplane it drives 8 GbE links using fabric A, plus the IPMB management bus. - PCB3 contains the uplink ports and clock management. It also contains some Switch Management Interface (SMI) channels, using fabric D. SMI is 8b10b 125 Mb/s data used, for instance, to transmit UTC. AMCs can use the clock distributed in the TCLKA lines to decode this information. - PCB2 is used for distributing a corrected clock to the backplane using the TCLKA lines. - PCB4 is devoted to the remaining SMI channels.

Backplane interface MCH Design The signals used in the backplane include: - Fabric A for the GbE links. - Fabric D for the SMI links. - Both IPMB busses: the one to manage AMCs and the one for power blocks and fans. - TCLKA to distribute a phase-compensated 125 MHz clock.

MCH main board MCH Design The main switching function resides in the Cyclone 3 main FPGA, helped by a low-latency external RAM for MAC-matching lookup. For more complicated things (RSTP, SNMP, etc.) we use an embedded Linux system hosted in the ARM9 CPU. This includes a PTP daemon which has recently demonstrated PTP compliance in a plugfest (ISPCS 2010). The ARM7 serves in principle as a watchdog (to avoid ever losing contact with the system) and also as an I2C hub for implementing management through IPMI (not done and not needed for the WR switch). It should be said that this MCH is not a general-purpose high-power MCH (see e.g. the 64MB of RAM for the embedded Linux). It is rather a design tailored to fit the needs of the WR switch. This version (v2) of the MCH is the one we are working on for VHDL and software development so as to have a basic-switching implementation by the end of 2010.

Conclusion μTCA a good choice for our problem. Hardware debugged. and outlook Conclusion μTCA a good choice for our problem. Hardware debugged. PTP working. Currently working on VHDL and embedded Linux driver + user space tasks. Self explanatory

Outlook Basic switching (using v2 hardware) expected in December. Conclusion and outlook Outlook Basic switching (using v2 hardware) expected in December. MCH v3 in technical spec phase. Should be Virtex-6 based with no external PHYs. Self-explanatory More information at http://www.ohwr.org/projects/white-rabbit

Reserve material

Boot process MCH Design The watchdog CPU uses a challenge-response mechanism to make sure the main CPU is doing well. If not, it reboots the main CPU with a safe image.