1. A Multi-Paradigm Verification Flow Dave Whipp

2. Copyright © NVIDIA Corporation 2005 Multiple Paradigms Analog Mixed Signal Digital Formal Semi-formal Simulation Emulation Directed tests Directed random Constrained random Self checking Model based Cycle based Transaction based Unit level System level Plan driven Coverage driven Pure Verilog SystemVerilog VHDL SystemC E / Vera HDCaml / Confluence

3. Copyright © NVIDIA Corporation 2005 A Multi-Paradigm Verification Flow What do I mean by Multiple Paradigms? Why would we want one? Engineering a [ Multiple Paradigm ] Flow Was it worth the effort?

4. Copyright © NVIDIA Corporation 2005 Two Philosophies Conformity Emphasize Standardization Diversity Emphasize Innovation Synergy or Conflict?

5. Copyright © NVIDIA Corporation 2005 Context Large Project Multiple Teams Long Legacy People move between projects benefit of familiar environment exposure to alternative approaches

6. Copyright © NVIDIA Corporation 2005 Understanding Variation To understand what to standardize: you need to understand what not to standardize Personal Preferences Technical Aspects of the Designs Supporting Legacy Seeking the Next Big Thing

7. Copyright © NVIDIA Corporation 2005 Personal Preferences Emacs! vi

8. Copyright © NVIDIA Corporation 2005 Personal Preferences Choice of editor doesn’t affect others At least, not much Choice of scripting language has greater impact But is encapsulated A script’s users don’t see the implementation language Choice of HVL affects whole team Can’t write “E” tests for a “Vera” testbench! But a unit testbench isn’t seen by other units A good flow will allow encapsulation of preferences I can go to any unit and build & run its tests Enables rapid localization of infrastructure issues

9. Copyright © NVIDIA Corporation 2005 Technological Differences

10. Copyright © NVIDIA Corporation 2005 Technical Characteristics PCIEPCIE Video Frame Buffer Off-chip Memory Graphics Pipe

11. Copyright © NVIDIA Corporation 2005 Unit Level System Testing Stage N-1 Stage N+1 Stage N Graphics Pipeline (Transaction Model) RTL Unit DIFF

12. Copyright © NVIDIA Corporation 2005 Transactions Vs Cycles Data min_val (Addr a1, Addr a2) { Data d1 = mem_read(a1); Data d2 = mem_read(a2); if (d1 < d2) return d1; else return d2; } Address Data a1 d1 Pipelined Bus a2 d2 t1 t2 t3 t4 t5 t6

13. Copyright © NVIDIA Corporation 2005 Reuse Vs Stagnation Reuse considered Good Avoid reinventing the wheel Build on the shoulders of giants Reuse invites Inertia Reuse can propagate dependencies Dependencies make things harder to change Resistance to change is known as inertia Inertia can lead to Stagnation Improper reuse accumulates dependencies Reused code that is not understood will bit-rot To avoid stagnation, inject agitation

14. Copyright © NVIDIA Corporation 2005 New Challenges New Tools New Platforms New People New Ideas Refactoring Testability D.R.Y. Avoiding Stagnation

15. Copyright © NVIDIA Corporation 2005 Are Single Paradigm Projects Possible? Project 1 Unit A Unit B Unit C Project 2Project 3 Unit D Unit B Unit C Unit B Paradigm 1 Paradigm 2 Paradigm 3 time

16. Copyright © NVIDIA Corporation 2005 1999 GeForce 256 22 Million Transistors 2002 GeForce4 63 Million Transistors 2003 GeForce FX 130 Million Transistors 2004 GeForce 6 Series 222 Million Transistors 1993 NV1 1 Million Transistors 2005 GeForce 7 Series 302 Million Transistors A Rich Legacy A project must support at least three approaches Legacy Mainstream Future

17. A Paradigm-Agnostic Verification Flow

18. Copyright © NVIDIA Corporation 2005 What is a Verification Flow? The User Interface to Verification Command Line Interface Graphical Interface Web (browser) Interface Programmers Interface Used By Verification Engineers Designers Managers Robots

19. Copyright © NVIDIA Corporation 2005 Purpose of a Flow Conceptual Framework Where do things live? How do things work? Keep out of the way Don’t make life difficult for people Define Metaphors Subtly influence future directions Anchor for Variation Give people something to complain about

20. Copyright © NVIDIA Corporation 2005 Concepts A well defined interface behaves as expected Obvious how it works, just by looking (“affordances”) As consistent as possible, but no more so Options are decision points Users don’t read the manual

21. Copyright © NVIDIA Corporation 2005 Designing a Flow (Interface) Invent some Users Figure out the important activities Create the user model Sketch a first draft Iterate Watch some real users; and iterate again

22. Copyright © NVIDIA Corporation 2005 Invent some Users TimNalu Dusk Mike

23. Copyright © NVIDIA Corporation 2005 Figure out the important Activities Timbury New Hire Needs to get started Checkout Tree Run Sanity Regression Simple Debug Dusk Lead Designer Intensive Debug Modify RTL Quick Regressions Analyze Coverage Uses Emacs Nalu Testing Guru Creates Testbenches Hacks the Flow Intensive Debug Setup Regressions Uses vi Mike Manager View Progress Tests Written Pass/fail counts Coverage Windows ® User

24. Copyright © NVIDIA Corporation 2005 Task Summary build testbenches run quick regression tests (sanity tests) run specific test rerun specific test with debug options rerun specific phase(s) of a test Add a test Add a testbench overnight regression testing Summarize results (web page)

25. Copyright © NVIDIA Corporation 2005 User Model for Running Tests Build Testbench Generate Tests Run Predictor Run DUT Compare Behaviors

26. Copyright © NVIDIA Corporation 2005 The Implementation “Make” driven flow Makefiles define the build Automated testbench creation Compile/generate tests where appropriate “makepp” is a better “make” Perl scripting for the guts Each test is a Perl object Small number of standard classes Individual units define additional classes

27. rdistnvmkrun_unit GEN DIFF VSIM CSIM tbgen fmtbgen fmgen run_testgen compare “grok” randomize testlist rand_regress mod stgen ttgen build_testlist tjob LSF vcs An Early Sketch

28. Copyright © NVIDIA Corporation 2005 Command Line Interface nvmk nvmk sanity nvmk regress nvmk all nvmk open_bugs TESTNAME=bug42 nvmk open_bugs TESTNAME=bug42 VSIM=1 DUMP=1 nvmk open_bugs TESTNAME=bug42 CSIM=gdb

29. Copyright © NVIDIA Corporation 2005 Detailed Triage and Debug cd rundir/open_bugs/bug42 ls *.log gen.logcsim.log vsim.logdiff.log replay vsim.log +foo +bar replay –gdb csim.log

30. Copyright © NVIDIA Corporation 2005 Testing the Flow Considered a hard problem “If it works, then it works” Testability is a feature Start testing early Some tests are better than none Create a multi purpose “Example Project” A stable testing environment Acts as documentation Use the “Invented Users” to create activity scripts Keep it up-to-date

31. Copyright © NVIDIA Corporation 2005 Watch some Real Users Image courtesy of DAZ Productions

32. Copyright © NVIDIA Corporation 2005 How to “Watch” Users Meetings and Discussions Coffee-Break Grousing Bug Reports Keep Track of Support Requests create FAQs VNC (Remote Desktop) Instrumentation

33. Copyright © NVIDIA Corporation 2005 Biggest Problems Performance make-driven user interface was subjectively slow too much dependency checking users know what file’s they’ve changed how to communicate this to the tools? Too much flexibility Options imply decisions Lack of sufficient understanding So pervasive cut & paste reuse

34. Copyright © NVIDIA Corporation 2005 Build Time Distribution: 10,000 per sample

35. Copyright © NVIDIA Corporation 2005 Build Time Distribution: 1000 per sample

36. Copyright © NVIDIA Corporation 2005 Summary Verification Engineers do more than write tests A flow is a user interface Encourage user feedback Add instrumentation A flow is an enabler, not disabler encourage innovation don’t tell users what not to do Without diversity, a flow stagnates Without standardization, a flow flounders