Reconfigurable Computing Leveraging FPGA Accelerators in High-Performance Computing Applications Craig Ulmer June 2, 2005 Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL Craig Ulmer, Ryan Hilles, and David ThompsonSNL/CA Keith Underwood and K. Scott HemmertSNL/NM

What is Center 8900 doing with FPGAs? Computer Sciences & Information Technologies 8900 –High-Performance Computing –Viz & Scientific computing (8963) FPGA LDRD for accelerating HPC –NM:Computational aspects –CA:Communication aspects FPGA LDRD for network security –NM/CA: Smart routers Catalyst

Outline Reconfigurable Computing –Communication Aspects –Computational Aspects New Platforms: Cray XD1 Summary

In a nutshell.. Reconfigurable Computing –Use FPGAs as a means of accelerating simulations –Implement key computations in hardware: soft-hardware double doX( double *a, int n) { int i; double x; x=0; for(i=0;i<n;i+=3){ x+= a[i] * a[i+1] + a[i+2]; … } … return x; } * + + a[i]a[i+1] Z -1 a[i+2] SoftwareHardwareSynthesized Hardware FPGA CPU RC Platform

Trivial Example: Sorting Sort a matrix of 64b values Hardware implementation –Sorting element –Array of elements –Data transfer engine Exploits parallelism –Do n comparisons each clock –Requires 3n clock cycles Limited to n values –Use as building block Sorting Array Address Wr Req Rd Req DMA Engine Sorting Control Double-Buffered Outgoing Memory Reg > Sorting Element unsortedsorted

Sorting Performance Xilinx Virtex II/Pro50-7 FPGA –128 elements –110 MHz Data transfer by FPGA –Read/write concurrency –Pinned memory 2-4x Rate of Quicksort(128) –Host is 2.2GHz AMD Opteron

RC: Communication Aspects

New FPGAs are “Network Ready” Multi-gigabit transceivers (MGTs) –V2Pro:twenty-four 3Gbps channels –V2ProX:twenty 10Gbps channels –InfiniBand, FC, GigE, 10GigE Multiple embedded processors –PowerPC MHz Direct connection to network –Build NI in hardware –Use PPC core for complex protocols –User-defined computational circuits Xilinx Virtex II/Pro-7 Network Interface PowerPC CPU User- Defined Circuits

TCP/GigE Network Interface for FPGAs Network interface (NI) in hardware –TCP Offload Engine(TOE) –Gigabit Ethernet(GigE) Refined TCP functionality –Slow start, NACK detection, Throttling… –User data rate to host: 600 Mb/s Outgoing TCP Message Control Rocket I/O TxRx Align Decode ARP Cache Framer MAC Header Control ARP Reply Ping ReplyIP Header Incoming Byte Stream Timeout Monitor CRC Outgoing Byte Stream Incoming TCP Message Control TOE GigE UnitSlicesV2P20 TOE2,06822% GigE1,10212%

Transport Framework for Visualization Need a framework for supporting Visualization in FPGAs –Want to perform viz tasks, not rendering –Built Chromium interface to transport OpenGL graphics Results –Functional system where visualization in FPGA, Rendering in Host –Saturate GigE link with OpenGL data Viz App GigE Network GigE TCP Offload Engine Chromium Packetizer FPGA

Network Intrusion Detection System (NIDS) Collaboration with Georgia Tech –GT:Pattern matching cores –SNL: GigE NI cores Network Intrusion Detection System –Examine packets & remove malicious –Based on Snort rule set (1,500 rules) Filter multiple GigE links transparently –Demonstrated for two links NI Malicious Packet Pattern Matching NI OKDrop Scheduler

RC: Computational Aspects

Floating Point Computations Floating point has been weakness for FPGAs –Complex to implement, consume many gates Keith Underwood and K. Scott Hemmert –Developed FP library at SNL/NM FP Library –Highly-optimized and compact –Single/Double Precision –Add, Multiply, Divide, Square Root –V2P50: DP cores, MHz

Example: FP Distance Computation Compute c=sqrt(a 2 + b 2 ) –Simple pipeline (88 stages) Implemented on Xilinx Virtex II/Pro 50 –159 MHz clock rate –39% of V2P50 –75 MiOperations/s Incoming SRAM Outgoing SRAM DMA Engine Incoming SRAM From Host To Host

New Platforms: Cray XD1

Cray XD1 Dense multiprocessor system –12 AMD Opterons on 6 blades –6 Xilinx Virtex-II/Pro FPGAs –InfiniBand-like interconnect –6 SATA hard drives –4 PCI-X slots –3U Rack

Cray XD1: Placing FPGAs Near Memory Blades use HyperTransport –High bandwidth I/O FPGA accelerator –Xilinx Virtex II/Pro 50 (-7) –HT connection: GB/s –10x performance of PCI First of a new breed –Commodity HT accelerators CPU Main Memory Main Memory NI SRAM Expansion Module SRAM FPGA NI 1.6 GB/s HT Processor Blade

Summary FPGAs are getting faster and less expensive –Finally have capacity for interesting problems –Million gate parts for $5 Large amount of reconfigurable computing work at SNL –Experience with Virtex II/Pro Platform FPGAs –Networks & embedded CPUs –Willing to share cores, work with others

Deliverables NetworkingVisualizationComputationsXD1General OpenGigE OpenTOE Raw GigE GigE Bridge NIDS Filter Isosurfacer Chromium Packetizer Sorting Array MD5 2D Distance DMA Engine Computation Framework Asynch. Message FIFOs Clock domain data pass PowerPC instantiation SNL/CA Cores 32b/64b Floating-PointComputations Add Multiply Divide Square root Vector/Matrix Multiply SNL/NM Cores

FPGA Slices Relative V2P Price & Density V2P100 V2P70 V2P7 V2P40 Relative Price & Density