June 2007 RAMP Tutorial BEE3 Update Chuck Thacker John Davis Microsoft Research 10 June, 2007

June 2007 RAMP Tutorial Outline What is BEE3? BEE3 board properties BEE3 gateware BEE3 schedule

June 2007 RAMP Tutorial What is BEE3? Follow-on to BEE2 (BWRC, 2004) Board with several highly-connected FPGAs Vehicle for computer architecture research –Microsoft’s primary interest Potential platform for high performance DSP applications –Astronomers, and perhaps others. Allows large scale architectural experiments –Although perhaps not as large as originally hoped –And certainly not at the speed of a real implementation Can scale smoothly from a single board to 64 boards (256 FPGAs)

June 2007 RAMP Tutorial BEE2

June 2007 RAMP Tutorial BEE2 – BEE3 Differences 4 Xilinx Virtex 5 vs 5 Virtex 2 Pro FPGAs –We use XC5VLX110T-ff1136 –V2Pro is now obsolete (130nm) –V5 is a major improvement (65nm) 6-input LUT (64 bit DP RAM) Better Block RAMs Improved interconnect Better signal integrity 8 Infiniband/CX4 channels vs 18 4 x8 PCI Express Low Profile slots

June 2007 RAMP Tutorial BEE3 – BEE2 Differences (2) 2 Banks DDR2 x 2 vs 4 Banks DDR2 x 1 –64 GB capacity with 4GB DIMMs –Lower total bandwidth, but higher per-channel rate 500 MT/s vs 400 –Mandated by fewer signal pins on V5 4 10/100/1000 Ethernet channels No PowerPCs –This version has not yet been released by Xilinx When it is, we can use it

June 2007 RAMP Tutorial BEE2 – BEE3 Differences (3) Divided the system into two boards, Main and Control –Main board has FPGAs, all high speed logic –Control board handles downloading, monitoring Being designed at BWRC –Simplifies main board engineering – can design control board in parallel –Initially, will have a simplified control board. System bring-up uses JTAG. Smaller main board –211 vs 374 in 2 –Fewer layers for lower cost Much more “PC-like” Fewer on-board peripheral interfaces –Those that are there will work Uses PC power supplies, peripherals Schematic is complete, layout is in progress –Fits in 2U enclosure –Much more attention is being given to thermal design –Must pass UL, FCC

June 2007 RAMP Tutorial BEE3 Main Board

June 2007 RAMP Tutorial BEE3 Board Layout

June 2007 RAMP Tutorial BEE3 Package

June 2007 RAMP Tutorial BEE3 Package Front View

June 2007 RAMP Tutorial Bandwidths (per-FPGA) Memory –500 MT/s * 9B/T * 2 channels: 9.0 GB/s Ring –500 MT/s * 9 B/T * 2 channels: 9.0 GB/s QSH –400 MT/s * 10 B/T: 4 GB/s Ethernet –125 MB/s CX4 –1.25 GB/s * 2 directions * 2 channels: 5GB/s PCI Express –Same as CX4

June 2007 RAMP Tutorial Initial Gateware Mostly things required for production testing and board characterization: Signal connectivity checks –Some are AC coupled, must test at speed –QSH tested with a crossover card Temperature and power supply monitoring DDR-2 Controller –Useful in other designs CX4 and PCI Express for at-speed tests Xilinx and others have lots of IP

June 2007 RAMP Tutorial Project Participants and Roles Microsoft Research (Silicon Valley) –Funds, manages system engineering, does some gateware Celestica (Ottawa and Shanghai) –Does main board engineering, prototype fabrication –Microsoft has a very deep relationship with Celestica TBD (Maybe Celestica, maybe ???) –Builds and delivers functioning systems Function Engineering (Palo Alto) –Does thermal and mechanical engineering Xilinx (San Jose) –Provides FPGAs for academic machines (slowest grade) –Provides FPGA application expertise Ramp Group (BWRC) –Control board, basic software Ramp Community –Uses the systems for research –Expanding to industrial users (e.g., us)

June 2007 RAMP Tutorial Schedule Generate Specification – Done Schematic Entry – Done Board Layout – Started Thermal modeling, heat sink design – Started Chassis design -- Started Signal Integrity – Imminent Prototypes: Late Summer – Bring-up starts Production: Start winter ‘07

June 2007 RAMP Tutorial Why is Microsoft interested? We believe the overall RAMP effort can have significant impact, and want to support it in the most effective way we can. –Simply paying for grad students seems suboptimal We observe that universities aren’t very good at this sort of system engineering and production. –Grad students are great for many things, but doing things like board layout aren’t among them. –Requires deep understanding of tools and production processes. Pros have this. –We can open doors that academia can’t. –We have experience in managing this sort of program. We want the systems ourselves –As infrastructure for our new effort in computer architecture (yes, this is a recruiting pitch). We also want systems to be available to other industrial users –This might be more difficult if the systems came from academia. –But we don’t want to be in the hardware business.