Intelligent Interleaving of Scenarios: A Novel Approach to System Level Test Generation Shady Copty, Itai Jaeger(*), Yoav Katz, Michael Vinov IBM Research Laboratory in Haifa

1 Outline System-level stimuli generation challenges System-level stimuli generation challenges Interaction based testcase generation Interaction based testcase generation X-Gen: a system-level testcase generator X-Gen: a system-level testcase generator Scenario interleaving in X-Gen Scenario interleaving in X-Gen Real life experience examples Real life experience examples – eServer, Cell BE, Xbox CPU chip Summary Summary

2 System verification System verification is aimed at validating the integration of several previously verified cores in a relatively short time

3 Challenges in Stimuli Generation for System Level Functional Verification Specifying system level scenarios in an abstract form Specifying system level scenarios in an abstract form –While generating the required low level stimuli Generating coordinated system-level stimuli Generating coordinated system-level stimuli –Projected to each and every core in the system Effectively handling configuration changes Effectively handling configuration changes –2-way system vs. 8-way system Adapting to core modifications and new cores Adapting to core modifications and new cores –PCI  PCIe

4 The concept of interaction Capturing the essence of the system-level functionality Capturing the essence of the system-level functionality –CPU-to-memory read/write –CPU-to-CPU inter processor interrupt –CPU initiated MMIO –IO initiated DMA –… Interaction: acts, actors Interaction: acts, actors Independent of the system’s configuration Independent of the system’s configuration Agnostic to specific core details Agnostic to specific core details

5 A DMA Interaction The concept of interaction - example  A CPU stores to the doorbell register of the DMA engine PLB CPU#4 CPU#3 CPU#2 CPU#1 DMA Engine Bridge Interrupt Controller Memory PHB IO BFM#2 IO BFM#1  The data is moved from the IO port to memory  The DMA engine interrupts the initiating CPU

6 Interaction based scenarios Stress the system bus through address contention on requests from multiple devices Verification goal Generate multiple IO reads, DMAs, and processor accesses to the same address Test plan definition Generate contention on the system bus Interaction based scenario

7 X-Gen: a model-based system- level stimuli generator Test Template *.rqst X-Gen Engine Interactions (Transactions) Configuration (Topology) Components (Cores) Abstract System Model Test Case CSP Solver Specifying an interaction-based scenarios

8 Interaction as the basic building block Interaction as the basic building block Control over: Control over: –Participating cores –Interaction properties A hierarchy if higher level statements A hierarchy if higher level statements –One-of: weighted random choice –Repeat –All-of Request file All of Repeat x10 Load / storeDMAinterrupt Target: mem2Addr: 0x12??Priority: One of Weight: 70Weight: 30 A system-level test template

9 A system-level test case Processors Instructions Registers initializations Translation tables Memory Data initializations Behaviorals (BFMs) Internal memory initializations Commands, such as: Send an interrupt Initiate a DMA Hubs, bridges, adaptors Translation tables Data descriptors for send / receive operations Internal registers initializations PLB CPU#4 CPU#3 CPU#2 CPU#1 DMA Engine Bridge Interrupt Controller Memory PHB IO BFM#2 IO BFM#1

10 Combining scenarios Stressing the system through parallel execution of several system level scenarios Stressing the system through parallel execution of several system level scenarios For example: For example: –Read/write cache coherency from multiple processors –MMIO / DMA to IO port –Inter processor interrupts –Create address contention between DMAs and processor accesses –Send interrupts to processors contending on the same cache line

11 CPU_to_memory_data_tarnsfer IO_to_memory_DMA CPU_to_memory_data_tarnsfer IO_to_memory_DMA … Scenario interleaving in X-Gen CPU.rqst - Repeat 80 - CPU_to_memory_data_tarnsfer IO.rqst - Repeat 70 - IO_to_memory_DMA X-Gen Total: 150 interactions Accessing the same address

12 eServer system verification: Partitioned system-level test template library Cache coherency scenarios PCI bridge stress scenarios InfiniBand message passing scenarios Intervention scenarios X-Gen Maintenance saving: avoiding exponential test template growth

13 Xbox 360 chip verification : Reusing scenarios eServer MP/MT scenarios Xbox specific IO scenarios X-Gen Heavy vIP reuse: generating complex system-level testcases within weeks from verification launch

14 General cell scenarios Power management function Performance management function X-Gen Main scenario Power down unit XXX Main scenario Restart unit XXX Main scenario … Cell BE verification: Verifying pervasive functions Main scenario Slow down unit YYY Main scenario Power down unit XXX Main scenario Resume frequency on unit YYY Restart unit XXX Main scenario …

15 Summary Interleaving transaction-level scenarios is crucial for system-level verification Interleaving transaction-level scenarios is crucial for system-level verification Partitioning the test plan according to different system functions Partitioning the test plan according to different system functions –Avoids costly maintenance of test templates –Promotes reuse of verification IP –Fast and effective verification of pervasive functions in conjunction with “mainstream” scenarios Implemented in X-Gen Implemented in X-Gen –IBM’s system-level testcase generator –Used in the verification of eServer, Xbox & Cell systems

Thank You !!!

17 Traditional Approach #1: Combining Lower Level Drivers Advantages Advantages –Simple –Quickly adapts to new cores –Reuse of core VIP Disadvantages Disadvantages –Not system level verification No coordinated stimuliNo coordinated stimuli No system level scenariosNo system level scenarios IP1 Driver IP2 Driver IP2 Driver BUS

18 Traditional approach #2: Transaction Based Verification Advantages Advantages –System level abstraction –Allows complex coordinated stimuli –Allows reuse of test specification Disadvantages Disadvantages –Transactor code is monolithic –Difficult to adapt to configuration changes –Difficult to adapt to new components / changed functionality IP1 Transactor BUS IP2 Driver Transactor

19 Request files: a dual effort methodology Specifying a scenario Interactions as building blocks Interactions as building blocks Restrict actors, properties Restrict actors, properties Inter-interaction relations Inter-interaction relations Request file All of Repeat x10One of DMA transfer CPU load / store Interrupt Read: 80 Write: 20 Address Collision: 65% The Bug Intelligent background noise Built-in testing knowledge Built-in testing knowledge User direction User direction