1. RAMP Retreat Summer 2006 Break Session Leaders & Questions Greg Gibeling, Derek Chiou, James Hoe, John Wawrzynek & Christos Kozyrakis 6/21/2006

2. Breakout Topics RDL & Design Infrastructure RAMP White Caches, Network & IO (Uncore) RAMP2 Hardware  BEE3 OS, VM and Compiler  Software Stack

3. RDL & Design Infrastructure Leader/Reporter: Greg Gibeling Topics  Features & Schedule  Proposals Multi-platform migration  Languages Which languages, priorities Assignments for support  Debugging – Models & Requirements  Retargeting to ASICs (Platform Optimization)

4. RDL & DI Notes (1) Languages  Hardware Verilog BlueSpec IBM uses VHDL  Software? Multi-Platform  Integration of hardware simulations  Control of multiplexing Needed for efficiency! Possible through channel & link parameters Features  Meta-types  Component (and unit) libraries

5. RDL & DI Notes (2) Debugging  Split target model RDL Target Design Exposed to a second level of RDL Allows statistics aggregation Modeling of noisy channels  Integration with unit internals Event & State Extraction Connection to processor debugging tools  People clearly want this ASAP

6. RDL & DI Notes (3) Debugging (Integrated)  Message tracing Causality Diagrams Framework to debug through units  Checkpoints  Injection  Single stepping May not be widely used But cheap to implement  Watch/Breakpoints

7. RDL & DI Notes (4) Why Java?  Runs on various platforms Recompilation is generally pretty painful  Decent type system in Java 1.5  Perfect for plugin infrastructure (e.g. OSGi) When to use RDL  Detailed timing model  Great at abstracting inter-chip comm  Perfect platform for partitioning designs Concise, logical specification  Support for the debugging framework  With standard interfaces, good for sharing

8. RDL & DI Notes (5) Basic Infrastructure  First system bringup Interfaces with workstations Initial board support  Standard interfaces (RDL and otherwise) Processor Replacements Board Support  Currently a heroic effort  Solutions Standardized components? Generators?

9. RDL & DI Notes (6) Timelines  Greg’s Goals 10/2006 should see RCF/RDLC3 11/2006 should see documentation  Debugging (Integrated) should be ASAP Manpower  Board support  First board bring up  RDL & RDLC users Standard interfaces Features & Documentation

10. RAMP White Leader/Reporter: Derek Chiou Topics  Two day break-out First day should be pro/con  Overall Preliminary Plan Evaluation Who is doing exactly what?  ISA for RAMP White OpenSPARC 32bit Leon PowerPC 405 Processor agnosticism  Implementation Reimplementation will be required Test suites from companies are very useful

11. RAMP White Notes (1) Use embedded PowerPC core first  Available  Debugged  Can run full OS today  FPGA chip space is already committed PowerPC and Sparc are both candidates PowerPC pros  Embedded processor is PowerPC Sparc pros  64b available today Wait and see on soft-core for RAMP-White from Derek go here

12. RAMP White Notes (2) >= 256 processors  Can buy 64 processors today Reasonable speed  10’s of MHz With 280K LUTs in Virtex 5, assume 50% for processor but 80% for ease of place- and-route  100K LUTs for processors  Need 4 per FPGA (16 per board, 16 boards)  25K LUTs per processor

13. RAMP White Notes (3) Embedded PowerPC core (it’s there and better performance than any soft-core)  Soft L1 data cache (no L2)  Hard L1 instruction cache  Emulation???? Ring coherence (a la IBM) Linux on top of embedded PowerPC core NSF mount for disk access Mark’s port of Peh’s and Dally’s router To do:  Ring coherence + L1 data cache + memory interface  RDL for modules  Software port  Timing models for memory, ring, cache, processor?  integration

14. RAMP White Notes (4) RAMP-White Greek  Beta More general fabric using same router Still use ring coherence  Gamma James Hoe’s coherence engine  Delta Soft core integration

15. Caches, Networks & IO (Uncore) Leader/Reporter: James Hoe Topics  CPU, Cache and Memories  Hybrid FPGA Cosimulation  Network Storage  Interfaces Especially with respect to interfaces Components, not sub-frameworks  Phase uncore abilities

16. Uncore Notes (1) A fully-system has more than just CPUs and memory I/O is very important Getting RAMP to “work”  Just like the real thing (from SW and OS’s perspective)  Software porting/development  Performance studies Someone has to build the “uncore”?  Co-simulation  Direct HW support for paravirtualization / VM

17. Uncore Notes (2) Why make RAMP white generic?  What is a more interesting target system?  What is a more relevant target system? Building a system without an application in mind? Would anyone care about RAMP- “vanilla”?

18. Uncore Notes (3) Why insist on directory-based CC for 1000 nodes  Today’s large SMPs (at 100+ ways) are actually snoopy-based  Plug in 8-core CMPs, that is a 1000-node snoopy system (that the industry may be more interested it in)

19. Uncore Notes (4) Let’s ping down a reference system architecture (including the uncore)  minimum modules required?  optional modules supported?  fix standard interfaces between modules  RDL script for RAMP white?? Need more than a block diagram for RAMP white

20. Uncore Notes (5) Requests and Ideas for RDL  Compensate for skewed raw performance of components (for timing measurements) Large I/O bandwidth relative to CPU throughput Need knobs to dial-in different rates for experiments  Some form of HW/SW co-simulation  Built-in performance monitoring

21. Uncore Notes (6) Sanity Check  1000 processing nodes: no problem  I/O: we can fake it somehow  DRAM for 1000 processing node Not easy to cheat on this one

22. RAMP2 Hardware (BEE3) Leader/Reporter: Dan Burke & John Wawrzynek Topics  Follow up to XUP Should RAMP embrace XUP at low end? Inexpensive small systems  Size & scaling of new platform More than 40 FPGAs?  Technical Questions Reconsider use of SRAM DRAM Capacity Presence of on-board hard CPUs On-board interfaces (PCI-Express)  Project Questions Timelines  Definitely need one Packaging Pricing (Especially FPGAs)  Design for largest FPGA, change part at solder time?  Evaluation of Chen Chang’s Design

23. RAMP2 HW Notes (1) Follow-up to XUP  XUP has been useful to the project, particularly for early development efforts.  Xilinx will continue to design and support new XUP boards  No v4 version planned.  V5 version will be out Q2 next year.  For BEE3 can't really count on V5 FX in 2Q next year.  Perhaps use a separate (AMCC) powerPC processor chip.

24. RAMP2 HW Notes (2) Size and Scaling of new platform: Given potential processor core density issue, will need to plan on a system that can scale past 40 FPGAs. Better compatibility with new XUP is important:  ex: DRAM standard (better sharing of memory controllers)  USB use Cypress CY7300 for USB compatibility with Xilinx core. Our design and production of BEE3 is timed to the production of V5 parts. We need to better understand RAMP team schedule for RAMP white. Hope to be able to choose the package and have flexibility in part sizes and ideally part feature set. How about a daughterboard for FPGA (DRC approach)?

25. RAMP2 HW Notes (3) Technical Questions Reconsider use of SRAM: group thought SRAM is a bad idea. It is faster, smaller, simpler to interface to. Newer parts will make interfacing simpler. Faster not a big concern for RAMP. Smaller is a big concern. 8GB DDR2 DIMM modules on the horizon. A target will be 1 GByte/processor. Presence of on-board hard CPUs  Are hard cores in FPGAs useful (e.g. PPC405 in V2Pro)  Would commodity chips on PCB be useful (eg for management)

26. RAMP2 HW Notes (4) Enclosures:  Using a standard form-factor will help in the with module packaging. Need to look carefully at IBM blade center (adopted by IBM and Intel) ATCA is gaining momentum.  Power may be a problem Can we accomodate custom ASIC integration (perhaps through a slight generalization of the DRAM interface). What does Google do for packaging in their data centers? Is it racks of 1U modules?

27. RAMP2 HW Notes (5) Interesting Idea from Chuck Thacker: "Design new board based on need of RAMP White"!  Previously suggested by others Can we estimate the logic capacity, memory BW, network BW, etc.?

28. OS, VM & Compiler Leader/Reporter: Christos Kozyrakis Topics  Debugging HW and SW (RDL)  Phased approach Proxy, full kernel, VMMs, Hypervisor HW/SW schedule and dependencies  High level applications

29. Software Notes (1) RAMP milestones  Pick ISA  Deploy basic VMM  Deploy OS

30. Software Notes (2) VMM approach: use split VMM system (ala VMware/Xen)  Run full VMM on x86 host that allows access to devices  Run simple VMM on RAMP that communicates with host for devices accesses through some network  A timing model may be used if I/O performed is important  Should talk with Sun & IBM about their VMM systems for Sparc and PowerPC.  May be able to port a very basic Xen system on our own Questions  Accurate I/O timing with para-virtualization (you also need repeatability)  SW/system-level/IO issues for large scale machine may be more important than coherence Related Issue: Do we want global cache coherence in white?  Benefit vs complexity (schedule etc)

31. Software Notes (3) Separate infrastructure from RAMP  Example: RDL should not be tied to RAMP White Note: This is in progress with some current RDL applications  Same with BEE3 design work  Most of our tools are applicable to others

32. Software Notes (4) Debugging support: RDL-scope  Arbitrary conditions on RDL-level events to trigger debugging  Get traces of messages  Track lineage of messages Traceability, accountability, relate events to program constructs  Infinite checkpoints for instructions & data  Checkpoint support Swappable & observable designs  Single step Instruction, RDL, or cycle level Note: not always a commonly use feature Such features may attract people to RDL more than retiming  Note: This is already the case with current RDL applications

33. Software Notes (5) What our is schedule  What can we have up and running with 1 year?  Does it have to be RAMP white? Do we need to migrate RDL maintenance from Greg?  Note: The work should be spread out at least. Do we have enough manpower for this SW work?  Compiler, VMMs, Applications, etc…

34. Software Notes (6) Application Domains  Enterprise/desktop Full featured OS on all nodes Running a JVM is a big plus here Should be able to run webservers, middleware and DBs.  Embedded While eventually an app may directly control a number of nodes, it is easier to start with all nodes running the OS. The base design should allow all nodes to run the OS.  Easiest starting point for SW. Various researchers may decide to run the OS in a subset of nodes, managing the rest of them directly  A simple runtime with app-specific policies Common in embedded systems

35. Software Notes (7) A simple kernel for embedded systems should support  Fast remapping of computation  Protection across processes Emulation of attached disk  ISCSI + a timing model for disks RAMP VMM uses:  Attract VMM researchers (might require x86)  Our own convenience Get an OS running, access to devices etc  We may achieve (b) without (a) Some researchers will want to turn cache coherence off anyway!