Test and Verification Solutions128 October 2009 Test and Verification Solutions Improved time to market through automated software testing Mike Bartley, TVS

2 Test and Verification Solutions228 October 2009 Agenda The potential benefits & drawbacks of automation Compare with HW verif Can automation reduce TTM? Automating static & dynamic testing Deciding what to automate Choosing your automation tools Case study

3 Test and Verification Solutions328 October 2009 The potential benefits of automation Reduced costs Improved quality Improved employee motivation Improved TTM

4 Test and Verification Solutions428 October 2009 The potential drawbacks of automation Shelf-ware! Maintenance Skills base Missing failures

5 Test and Verification Solutions528 October 2009 HW verification The mechanics of complex test bench Driver Stimulus generator TestScoreboard Functional Coverage Design Under Test Monitor assert constraint addrdata Assertions Coverage

6 Test and Verification Solutions628 October 2009 The mechanics of finding a bug Design Under Test PropagateStimulate … … … … … … ….. Actual Results Expected Results Compare Observe

7 Test and Verification Solutions728 October 2009 Can automation reduce TTM? Getting test results sooner Catching bugs earlier Stabilising the software sooner Assessing the stability of the software

8 Test and Verification Solutions828 October 2009 Automating static testing Types of bug found A quick win Beware false failures Introduce filters slowly Developers not testers (check in checks, IDE) No testing of intent

9 Test and Verification Solutions928 October 2009 Automating dynamic testing Test plan Test design specification Test case specification Test procedure specification Test execution Test log Test incident report Test summary report

10 Test and Verification Solutions1028 October 2009 Deciding what to automate Complexity of the test environment The test level Frequency of execution Ease of automating pass/fail Test stability and repeatability

11 Test and Verification Solutions1128 October 2009 Choosing your automation tools Who writes test test? Matching skill set Separate orthogonal concerns Portability Ease of adding/banning tests Integration into configuration/build system Cost of the tool?

12 Test and Verification Solutions1228 October 2009 Case Study – Background Includes chips, accelerator cards, and software All work with industry-standard x86-based systems

13 Test and Verification Solutions1328 October 2009 Case Study – Background Low level host driver code –Code on host computer communicates with the chip on the board in the PCI slot Libraries of code –Runtime library –Mathematics acceleration libraries (ACML, MKL, …) Application code –Linpack, Montecarlo, …. Code running on the chip –This had parallel C extensions not supported by our chosen tool Software models of the hardware In-house compiler for the code running on the chip Mainly C and C++, some assembler

14 Test and Verification Solutions1428 October 2009 Case study – automation details Integrated into configuration/build system Automated mostly lower level tests Developed internally by SW Engineers Offshored manual tests Offshored some automated test development Integration of static analysis Test maintenance

15 Test and Verification Solutions1528 October 2009 Case study - results of automation Improved quality Shortened release schedules Release schedules hit Reduced TTM ROI?

16 Test and Verification Solutions1628 October 2009 Questions? Mike Bartley LinkedIn