1. Combining Simulators and FPGAs “An Out-of-Body Experience”
Eric S. Chung, Brian Gold, James C. Hoe, Babak Falsafi
{echung, bgold, jhoe,
SIMFLEX/PROTOFLEX

2. The RAMP full-system challenge
RAMP vision for studying systems w/ FPGAs
functional & cycle-accurate simulation
scalability, speed, & flexibility on FPGAs
full-system (run unmodified binaries & OS)   
I/O MMU controller
DMA controller
IRQ controller
CPU
CPU
Terminal
PCI Bus
Memory
Graphics card
Ethernet controller
SCSI controller
Disk
Disk
‘Full-sys’ RAMP will incur large effort yet, not all behaviors frequently used (e.g., I/O)
June 22, 2006
Eric S. Chung / RAMP 2006 Summer Retreat

3. Combining simulators & FPGAs
Simulators already provide full-system
 why not simulate infrequent behaviors (e.g., I/O devices)?
FPGA
Simulator
CPU
CPU
CPU
CPU
Memory
SCSI
Ethernet
Memory
SCSI
Ethernet
disk
disk
Advantages
avoid impl. infreq. behaviors  lowers full-sys FPGA development
low impact on scalability & perf. on FPGA
June 22, 2006
Eric S. Chung / RAMP 2006 Summer Retreat

4. Outline Motivation Migration Implementation status Conclusion
June 22, 2006
Eric S. Chung / RAMP 2006 Summer Retreat

5. Migration Target design 3 ways to map target object to host
FPGA
Simulator
“Target objects” ex: func or timing cpu 3 1 2
3 ways to map target object to host
FPGA-only Simulation-only Migratable
Migratable objects
switch modes between FPGA & simulator hosts
target behavior need not be 100% in FPGA mode
e.g., impl. 80% target behavior in FPGA, 100% in simulator 1 2 3
June 22, 2006
Eric S. Chung / RAMP 2006 Summer Retreat

6. Example CPU instruction stream
Migration example
Target-to-host mappings:
CPU = migratable
Memory = FPGA-only
Devices = SW-only
load
CPU
CPU
FPGA
Memory
SCSI
Example CPU instruction stream
CPU state transfer
Simulator
SCSI cmd
load
CPU
add
time
multiply
I/O SCSI cmd
add
Memory
SCSI
sub ..
disk
June 22, 2006
Eric S. Chung / RAMP 2006 Summer Retreat

7. Advantages FPGA Simulator Lowers development effort Fast & scalable
avoid bring-up of infrequent behaviors
migrate & validate ref. models from simulator
tailor impl. to workload (avoid rarely used instrs, good for CISC x86)
Fast & scalable
perf-critical objects on FPGA (eg, CPU, memory)
scalable for MPs  add migratable CPUs
FPGA
Simulator
CPU
CPU
CPU
CPU
CPU
CPU
Memory
SCSI
Memory
SCSI
disk
June 22, 2006
Eric S. Chung / RAMP 2006 Summer Retreat

8. Subtleties Simulator FPGA Objects separated in simulator/FPGA interact
examples: interrupts, DMA
handle by forwarding messages between FPGA/simulator
FPGA-only & SW-only mapped objects easy to locate
migrated objects require tracking
FPGA
Simulator
CPU
CPU
CPU
DMA
Memory
SCSI
Memory
SCSI
disk
Forwarded DMA
June 22, 2006
Eric S. Chung / RAMP 2006 Summer Retreat

9. Subtleties Cross-host interactions rare  low impact on FPGA perf.
Objects separated in simulator/FPGA interact
examples: interrupts, DMA
handle by forwarding messages between FPGA/simulator
FPGA-only & SW-only mapped objects easy to locate
migrated objects require tracking
Option 2: Forced migration
Option 1: Forwarded
interrupt
FPGA
Simulator
CPU
CPU
CPU
Interrupt
Memory
SCSI
Memory
SCSI
disk
Cross-host interactions rare  low impact on FPGA perf.
June 22, 2006
Eric S. Chung / RAMP 2006 Summer Retreat

10. Subtleties cont. Migration cost FPGA & simulator asynchrony
migrating object requires state copy
e.g., migratable CPU has registers & TLBs
FPGA-to-simulator latency & sim. time limits # migrations/instr
FPGA & simulator asynchrony
simulated time “ticks” at different rates in FPGA & simulator
must synchronize for deterministic replay & accurate device timing
June 22, 2006
Eric S. Chung / RAMP 2006 Summer Retreat

11. Outline Motivation Migration Implementation in progress Conclusion
June 22, 2006
Eric S. Chung / RAMP 2006 Summer Retreat

12. Implementation status
Target system
Sun Fire[tm] 3800 Server (up to 24-way)
UltraSPARC III ISA
Solaris 8
Proof-of-concept software-to-software migration
run 2 instances of Virtutech Simics
migration designed & tested in 2 weeks
can migrate on arbitrary behavior (e.g., ADD instruction)
June 22, 2006
Eric S. Chung / RAMP 2006 Summer Retreat

13. BlueSPARC core (in progress)
In-order SPARCV9 core
supports 144 out of 170 integer instr behaviors
supports partial MMU w/ I- & D-TLBs
goal: % of instrs & behaviors in target workloads
SPEC (mostly user-level), OLTP/DB2 (high TLB misses, 40% time in priv-mode)
CPI ranges 5 to 7 cycles
synth: 15k LUTs on Virtex-II Pro 30, 85MHz, 12MIPS (worst-case)
developed in Bluespec HDL, 6000L in 6 weeks
Core validation
run RTL in lockstep w/ Simics’s UltraSPARC simulation model
workload validation w/ SPEC, OLTP/DB2, OpenSPARC verif. suite
June 22, 2006
Eric S. Chung / RAMP 2006 Summer Retreat

14. Migration on FPGA (in progress)
Xilinx XUP Virtex-II Pro 30
Virtutech Simics
Simics UltraSPARC
BlueSPARC
PowerPC
Migration & message interface
DDR memory
Simulated target devices
ethernet
PowerPC functions
core & memory initialization from Simics checkpoints
facilitates migration for BlueSPARC
connects simulated devices to memory (e.g., SCSI DMA)
June 22, 2006
Eric S. Chung / RAMP 2006 Summer Retreat

15. Conclusion Contributions Future work We are ready for BEE2
virtualizes infrequent behaviors using simulation
simplifies full-system FPGA emulator, still fast/scalable
incremental validation from reference system
Future work
support migration in RDL?
adding cores + scaling across multiple FPGAs
We are ready for BEE2
Thanks! Questions?
PROTOFLEX/SIMFLEX (
June 22, 2006
Eric S. Chung / RAMP 2006 Summer Retreat