ROACH II Review Introduction. ROACH II Review Agenda ● Agenda: – ROACH I, what was right, what was wrong? – Why ROACH II? – The place of ROACH II – Specifications.

Slides:



Advertisements
Similar presentations
Digital RF Stabilization System Based on MicroTCA Technology - Libera LLRF Robert Černe May 2010, RT10, Lisboa
Advertisements

Digital FX Correlator Nimish Sane Center for Solar-Terrestrial Research New Jersey Institute of Technology, Newark, NJ EOVSA Technical Design Meeting.
June 2007 RAMP Tutorial BEE3 Update Chuck Thacker John Davis Microsoft Research 10 June, 2007.
BEEKeeper Remote Management and Debugging of Large FPGA Clusters Terry Filiba Navtej Sadhal.
January 2007 RAMP Retreat BEE3 Update Chuck Thacker Technical Fellow Microsoft Research 11 January, 2007.
XUP Virtex-5 Development System January XUP Virtex52 Introducing XUPV5-LX110T A powerful and versatile platform packaged and priced for Academia!
1 Design of the Front End Readout Board for TORCH Detector 10, June 2010.
Using FPGAs with Embedded Processors for Complete Hardware and Software Systems Jonah Weber May 2, 2006.
System Architecture A Reconfigurable and Programmable Gigabit Network Interface Card Jeff Shafer, Hyong-Youb Kim, Paul Willmann, Dr. Scott Rixner Rice.
THE CPU Cpu brands AMD cpu Intel cpu By Nathan Ferguson.
UCT Software-Defined Radio Research Group
XP Practical PC, 3e Chapter 17 1 Upgrading and Expanding your PC.
Open, Scalable Real-Time Solutions Pentium Core Solo or Duo 1.6 to 2 GHz or Single or dual-core Pentium support in hard real-time up to.
© Copyright Xilinx 2004 All Rights Reserved 9 November, 2004 XUP Virtex-II Pro Development System.
Huazhong Normal University (CCNU) Dong Wang.  Introduction to the Scalable Readout System  MRPC Readout Specification  Application of the SRS to CMB-MRPC.
DLS Digital Controller Tony Dobbing Head of Power Supplies Group.
SLAAC SV2 Briefing SLAAC Retreat, May 2001 Heber, UT Brian Schott USC Information Sciences Institute.
Tuesday September Cambridge1 GDCC “next replacement of the LDA” Franck GASTALDI.
PCIe Mezzanine Carrier Pablo Alvarez BE/CO. Functional Specifications External Interfaces User (application) FPGA System FPGA Memory blocks Mezzanine.
RiceNIC: A Reconfigurable and Programmable Gigabit Network Interface Card Jeff Shafer, Dr. Scott Rixner Rice Computer Architecture:
GBT Interface Card for a Linux Computer Carson Teale 1.
XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Dariusz Makowski, Technical University of Łódź LLRF review, DESY, 3-4 December 2007 Advanced.
J. Christiansen, CERN - EP/MIC
Open, Scalable Real-Time Solutions Intel Core 2 Duo to Quad processor up to 5 VIRTEX II Pro FPGA board RT-LAB, SIMULINK, RTW, XILINX SG.
2. Requirements Existing Z-Dok interface 10G Ethernet – SFP+ Free chips? Fast Memory  More QDRII+ Large Memory for transient buffer  DDR? External Processor.
Nov 3, 2009 RN - 1 Jet Propulsion Laboratory California Institute of Technology Current Developments for VLBI Data Acquisition Equipment at JPL Robert.
CASPER Packetised correlators Jason Manley CASPER workshop 2009.
Atacama Large Millimeter/submillimeter Array Expanded Very Large Array Robert C. Byrd Green Bank Telescope Very Long Baseline Array CASPER Workshop 2009.
BEE3 Updates June 13 th, 2007 Chuck Thacker, John Davis Microsoft Research Chen Chang UC Berkeley.
Mr. Daniel Perkins Battelle Memorial Institute Mr. Rob Riley Air Force Research Laboratory Gateware Munitions Interface Processor (GMIP)
Acquisition Crate Design BI Technical Board 26 August 2011 Beam Loss Monitoring Section William Vigano’ 26 August
SRS Trigger Processor Option Status and Plans Sorin Martoiu (IFIN-HH)
XLV INTERNATIONAL WINTER MEETING ON NUCLEAR PHYSICS Tiago Pérez II Physikalisches Institut For the PANDA collaboration FPGA Compute node for the PANDA.
Calculator in a Box Co. Lead designer: Bradley Phelps.
E. Hazen -- Upgrade Meetings1 AMC13 Project Development Status E. Hazen, S.X. Wu - Boston University.
E. Hazen1 MicroTCA for HCAL and CMS Review / Status E. Hazen - Boston University for the CMS Collaboration.
Eric Hazen1 Ethernet Readout With: E. Kearns, J. Raaf, S.X. Wu, others... Eric Hazen Boston University.
XFEL The European X-Ray Laser Project X-Ray Free-Electron Laser Dariusz Makowski, Technical University of Łódź, DMCS ATCA LLRF project review, DESY, 3-4.
Netherlands Institute for Radio Astronomy 1 ASTRON is part of the Netherlands Organisation for Scientific Research (NWO) Board Design Gijs Schoonderbeek.
Components of Mechatronic Systems AUE 425 Week 2 Kerem ALTUN October 3, 2016.
M. Bellato INFN Padova and U. Marconi INFN Bologna
AMC13 Project Status E. Hazen - Boston University
Summary Remaining Challenges The Future Messages to Take Home.
Test Boards Design for LTDB
AMC13 T1 Rev 2 Preliminary Design Review E. Hazen Boston University
“FPGA shore station demonstrator for KM3NeT”
E. Hazen - Back-End Report
Microcontrollers & GPIO
AMC13 Status Report AMC13 Update.
The UniBoard Generic Hardware for Radio Astronomy Signal Processing
ROACH3 Introduction Alec Rust SKA-SA 1 1.
The Train Builder Data Acquisition System for the European-XFEL
LFD Correlator MWA-LFD Kick-off Meeting San Diego 5-June-2006
MicroTCA Common Platform For CMS Working Group
Electronics for Physicists
Unit 2 Computer Systems HND in Computing and Systems Development
Design Package Contribution for Project Olympus US1-EPYC
Table 1: The specification of the PSICM and the ePSICM Prototypes
University of Illinois, at Urbana-Champaign
Introduction to Computing
Progress Update Henry Chen December 17, 2010.
XC9500XL New 3.3v ISP CPLDs.
ChipScope Pro Software
Odroid XU4.
Electronics for Physicists
ChipScope Pro Software
Wireless Embedded Systems
A Look at Computer Parts
Command and Data Handling
Raspberry Pi Hardware By: Mike Kwiatkowski.
Presentation transcript:

ROACH II Review Introduction

ROACH II Review Agenda ● Agenda: – ROACH I, what was right, what was wrong? – Why ROACH II? – The place of ROACH II – Specifications – Financial Implications – Block diagram – Mezzanine boards – Mechanical and Thermal – Tool flow – Applications – Downgrading – Schedule and plans – Risks – Open discussion – Action items

Looking back at ROACH ● Good: – Architecture ● External PPC ● Good mix of memory types ● Useful interfaces – Compatibility ● ATX, mechanically and power supply ● Z-DOK ● 10GbE – Appears to be reliable – In production

Looking back at ROACH ● Not so good – Lantronix Xport firmware undocumented and changes without notice – Actel Fusion hard to program, a few failures of on-board Flash – National 1GbE phy sometimes (~5%) fails in production – Production test system is a bit of a hack – Memory bandwidth somewhat less than potential Z-DOK bandwidth – Possible to brick the board during Flash update – need JTAG tool to bring back to life – Want more digital bandwidth for next generation samplers

Why ROACH II? ● It's the Casper way ● Moore's law ● Xilinx Virtex 6 ● Fast ADC's ● CX4 is not likely to grow ● meerKAT

IBOB ● Good ● Cheap ● Simple ● ZDOKs for ADC inputs ● SRAM ● Bad ● No external processor ● Small FPGA capacity ● SRAM low bandwidth ● Limited IO ● Custom housing and PSU ● Good for ● F engines ● spectrometers

BEE2 ● Good ● Lots of IO ● Lots of DRAM ● Bad ● Control FPGA wasted ● No ADC interface ● Expensive ● Reliability concerns ● DRAM hard to use ● Custom PSU and case ● Good for: ● X engines ● Beamformers

BEE3 ● Good ● Huge processing density ● Bad ● Expensive ● Closed-source ● No onboard processor ● Custom size ● poor IO relative processing power ● Good for ● ??

ROACH1 ● Good: ● External PPC ● Well-chosen FPGA ● QDR and DDR ● Health monitoring ● ATX mech compatible ● Bad ● Hard to manufacture ● CX4 dying standard ● Overly complicated ● Need JTAG for update ● Good For ● Just about everything! ● Correlators ● Beamformers ● Pulsar Machines ● Spectrometers ●...

MORPH Massive Open Reconfigurable Performance Hardware Morph Open Reconfigurable Performance Hardware ● Good for: ● Low-cost pre-processors (F engines or Spectrometers) ● IBOB replacement ● Good ● Low cost ● Low power ● Incorporates lessons learnt from ROACH1 ● Bad ● No QDR ● Limited DDR bandwidth ● Limited IO ● Not sure ● Student project ● Aggressive schedule

ROACH 2 ● Good ● All ROACH-1 wins ● Reduced complexity ● Increased performance ● Flexible IO (SFP+, CX4, RJ45) ● Careful performance matching of onboard resources. ● Bad ● Expensive?? ● Good for: ● Large systems

ROACH 2: Specifications Motivation ● Primary design towards MeerKAT ● Largest resource consumer: Correlator ● Proportion resources for this application. ● Beamformer low on compute resources, high on IO (depending on number of synth'd beams) ● Want to keep R2 as useful platform for pulsar work, spectrometers and other basic instruments. ● Maintain backwards compatibility with existing hardware. ● Increase compute density, but keep 1FPGA/board.

Correlator requirements summary ● Virtex-6 SX475 main FPGA. ● Four 36bit QDR parts of 36Mibit each (144Mibit for MeerKAT2 upgrade path). ● A 72-bit DRAM DIMM slot running faster than 250MHz DDR. ● At least four 10GbE ports (more will be needed for beamforming). ● Increased PPC-FPGA datarates (32bit bus?).

Logic: FPGA sizing ● Logic: As big as possible! ● More FFT channels. ● Longer X engines. ● SX-series has good ratio of BRAM/Logic ● Logic/BRAM budgets with SX475 ● 512-input correlators ● Up to 8K FFT channels at 4GHz bandwidth

QDR Memory bandwidth ● Bandwidth scale 2^N, like FFT. ● Big matrix transpose operations possible, and fast, simple VACC implementations. ● ROACH2's four 36bit QDRs appear in application space as four 72bit SDR interfaces with simultaneous read/write. ● Flexibility: can be used for low-bandwidth deep depth, or in parallel for high bandwidth. ● F engine can transpose an 8GHz (16Gsps) stream!

QDR Memory capacity ● ROACH2 can be populated with different size QDRs. ● With maximum current capacity of 4x144Mib QDR memories: ● Packetise up to 32Ki FFT channels for 512-input correlator (or two streams of 64Ki FFT channels for 256-input correlator). ● VACC of 256-input correlator and 64Ki FFT chans. ● Need more? Sure: populate with 288Mib QDR, or move onto DRAM!

DDR ● Used where large memory capacities required (circular buffering and VACC for high-res spectrometers and large correlators) ● Variable latency; prefer QDR where possible. ● DRAM DIMMs expensive (FPGA pin resources). ● Single DIMM sufficient.

10GbE output ports ● ROACH's mezzanine connector is 6.25Gbps capable. With -2 speed devices, RXAUI enables up to 16x 10GbE ports per device. ● Two mezzanine cards. Can mix one with four CX4s (to mate to IBOB/BEE2/ROACH1) and one with four or eight SFP+ ports (to mate to new switches or driving optical fibre). ● Matched for dual-pol 4GHz stream using std XAUI and slower speed-grade devices. ● Can digitise and output streams up to 16GHz (quantised) bandwidth.

Board Management Hardware Monitoring Revamp ● Actel Fusion replaced with various COTS ICs to perform same function ● No remote power management interface FTDI Debug Interface ● USB to Multi-purpose IO Chip ● 2xUARTs, JTAG, IIC, GPIO access from host PC

PowerPC ● V6 bus interface now 32-bits (330 MB/s) ● SelectMAP interface directly driven by Processor Bus ● DMA signals added to CPLD/MMC interface ● Marvell 88E1111 1Ge PHY replaces problematic National PHY ● Up to 1GB NOR flash ● No more programmable clocks

FPGA ● Designed for Xilinx Virtex-6 SX475T or LX550T ● LX240T, LX365T, SX315T can be used with some loss in functionality

FPGA Memory ● From 2x18-bit to 4x36-bit QDR II+ ● From DDR2 → DDR3 (Registered DIMM only)

FPGA External I/Fs ● 2 x ZDOKS – up to 1.25 Gbps per pair (from 1 Gbps) ● 2 x MGT Breakout Slots ● 4 x Ten Gigabit Ethernet per Slot ● Media independent – could support CX4/SFP+/??? ● Supports up to 6.25 Gbps (RXAUI) ● 6 Single-Ended GPIO for MDIO, etc. ● 12 Single-ended GPIO (LVCMOS15), 8 LSBs of connected to LEDs ● Single Gigabit Ethernet (Marvell 88E1111) ● 2 x Auxiliary SMA inputs, 1 SMA output

ROACH II Mezzanine Cards/Boards ● Dimensions L x W x H (mm) ● 103 x 90 x 92 (total incl connector pair) ● Connector by SAMTEC ● QMS/QFS High Speed ● Rated at 8GHz/16Gbps ● 3V3/1V5, 12GPIO, 4 x 32 transceiver pairs ● Issues ● SFP+ mounting ● CX4 mounting ● Combinations ● RFI ● Cooling Altium Schematic of SAMTEC connector 3D model of SAMTEC connector

3D Impressions of board possibilities, CX4 and SFP+

ROACH II Mechanical and Thermal ● Roach II enclosure to be 1U ● Same format as ROACH I with telescopic slides and customizable front face ● ISSUES thermal and mechanical ● Suitable enclosure manufacturer ● RFI with regard to SFP+ and cooling requirements (shielding effectiveness) ● Current Internal Air Flow not sufficient to cool PPC, ROACH I. PPC on ROACH II may require fan cooled heatsink. ● Air flow channeling will be challenging with the mezzanine cards. Rear panel vent space will be limited. ● Moving to Cfdesign 2010 with review by Blue Ridge Numerics. ● Water cooling

Thermal Vector plot of ROACH I

Power PC Temperature Input Ambient Temperature QDR TemperatureXylinx V5 FPGA Temperature

Exit Ambient Temperature

ROACH2 Toolflow ● Maintain compatibility with ROACH1. ● Hardware cores identical. ● Transparent port of existing designs. ● No complications like BEE2/iBOB → ROACH because Simulink primitives identical.

Pricing ● Xilinx V6 Costs: (1759 pin package) ● Avnet Express: V6 LX240T - $2, ● V6 LX550T - $8K-10K (likely 3 x more than LX240T) ● SX parts tend to be more expensive than LX ● V6 LX475T - $9K-12K (likely 4x more than 240T) ● Additional: 2xQDRs ($120 each), SFP+ (100$ per port) ● Board costs: $ 4000 (including SFP+) or $ 3200 (CX4) ● Total cost: Likely $ with SX475T

Downgrading ● ROACH II designed for SX475T or LX550T ● LX240T, LX365T or SX315T can also be used with losing the following: ● 1 Full QDR Memory, 18 Bits of a QDR ● 3 GTX tiles: lose 2 x 10Ge ports and SGMII port ● 4 x Single-ended GPIO

Risks ● Cost ● XC6VSX475T is likely to be 4x the cost of XC5VSX95T ● Availability ● Large devices not available yet... ● Technical ● Amongst others: – Mezzanine card adds connector to high speed IO – 45A on 1V – current density requires careful attention – General power and signal integrity

Schedule ● Schematics: Complete ● Layout: late June 2010 ● First prototype manufactured: mid July 2010 ● Initial bring-up and testing: late August 2010 ● First production run: September 2010