1. Test Specifications A Specification System for Multi-Platform Test Suite Configuration, Build, and Execution Greg Cooksey

2. Test Specifications -2- Talk Outline Test suite challenges and basis for our new architecture The specification system A walk through the new Dyninst test suite architecture Other features, status notes, and future plans

3. Test Specifications -3- Idealized Dyninst Test Mutator program –Uses Dyninst to analyze or modify mutatee Mutatee program –Simply verify that modification occurred Output –passed, failed, or skipped Simple, right? mutator Dyninst mutatee output test

4. Test Specifications -4- Not That Simple Multiple platforms –Architecture –operating system –operating system version –32 vs. 64 bit Multiple compilers –Native –GNU –Other Multiple languages –C–C –C++ –Fortran Multiple build parameters –Optimization levels –Stripped vs. unstripped Multiple test parameters –Create vs. attach Multiple mutator-mutatee mappings And not all combinations are valid

5. Test Specifications -5- Our Goal Write simple test cases: Modify specification file –Don’t modify code when adding a test Automatically do the rest –Build the test programs, with appropriate variations and parameters –Run the test with appropriate parameters –Collect results and insert into database mutator() { insertSnippet(…)... } foo() {... }

6. Test Specifications -6- New Infrastructure Specification language and compiler –Specifications for almost all aspects of test suite Test suite build system –Standard build system –Makefiles are output of spec. compiler Test driver –Tests are plugins –List of tests to run is output of spec. compiler Results parser and database User only needs to write test cases and test specifications

7. Test Specifications -7- Specifications What? –Platforms Architecture Operating system File name conventions Compiler availability –Compilers Language Standard flags Option-parameter string mapping –Languages –Test cases Mutator sources Mutatee sources Mutator-mutatee mappings Build configurations Run configurations –Test suite glue Standard libraries Constraints How? –We use simple tuples, written as Prolog clauses Prolog lets us naturally express relationships between tuples

8. Test Specifications -8- Test Suite Architecture Build Stage Specifications Run StageData Collection Test sources

9. Test Specifications -9- Test Suite Architecture Build StageRun StageData Collection spec file spec compiler Test sources

10. Test Specifications -10- Specification Compiler mutator specmutatee spectest specstandard libscompiler spec Prolog Specifications mutator tuple mutatee tuple test tuple Tuples Python makefile test lists Scripts test suite glue

11. Test Specifications -11- mutator(‘example’, [‘example.C’]). mutatee(‘example’, [‘example_mutatee.c’]). compiler_for_mutatee(‘example’, Compiler) :- compiler_language(Compiler, ‘c’); compiler_language(Compiler, ‘c++’). mutator_mutatee(‘example’, ‘example’). test_runs_everywhere(‘example’). test_runmode(‘example’, ‘both’). test_start_state(‘example’, ‘stopped’). Test Specification mutator source mutatee source valid compilers mutator-mutatee mapping build and run constraints

12. Test Specifications -12- Test Suite Architecture Build StageRun StageData Collection spec file spec compiler Test sources

13. Test Specifications -13- Test Suite Architecture Run StageData Collection mutator compiler mutatee compilers build scripts (makefiles) spec filespec compiler Test sources

14. Test Specifications -14- Build scripts Specification compiler generates makefile Mutator built as shared library plugin –example.so Multiple mutatees built with build parameters encoded into names –example.mutatee_g++_none –example.mutatee_g++_low –...

15. Test Specifications -15- Test Suite Architecture Run StageData Collection mutator compiler mutatee compilers build scripts (makefiles) spec filespec compiler Test sources

16. Test Specifications -16- Test Suite Architecture Data Collection mutator compiler mutatee compilers build scripts (makefiles) spec filespec compiler run scripts (test lists) test driver src Dyninst compiler test driver mutator.so mutatee1 mutatee2 mutatee3 Test sources

17. Test Specifications -17- Run scripts Prolog generates tests as tuples –(example, example.mutatee_gcc_low, createProcess, stopped) Python script creates lists of tests that are compiled into the test driver Test driver executes lists of tests and produces results output

18. Test Specifications -18- Test Suite Architecture Data Collection mutator compiler mutatee compilers build scripts (makefiles) spec filespec compiler run scripts (test lists) test driver src Dyninst compiler test driver mutator.so mutatee1 mutatee2 mutatee3 Test sources

19. Test Specifications -19- Test Suite Architecture mutator compiler mutatee compilers build scripts (makefiles) spec filespec compiler run scripts (test lists) test driver src Dyninst compiler test driver mutator.so mutatee1 mutatee2 mutatee3 output log parser database Test sources

20. Test Specifications -20- Collecting Output Test output “example –mutatee example.mutatee_g++_low” All tests passed Output is human readable Added to database –(example, g++, low, createProcess): passed

21. Test Specifications -21- Flexible Testing Structures Build each test as separate binary –High isolation, slow performance Build several tests into a single binary –Balance test isolation vs. performance

22. Test Specifications -22- Not Just Tests Specifications also used for –Platforms Specify architecture, operating system, filename conventions, compiler availability* –Compilers Specify languages*, option - command line parameter mapping, standard flags, platform availability* –Languages Specify filename conventions, compilers*

23. Test Specifications -23- Example Platform % Linux 2.6 i386 specification % paragraph platform(‘i386’, ‘linux’,‘linux2.6’, ‘i386-unknown-linux2.6’). compiler_platform(‘gcc’, ‘i386-unknown-linux2.6’). object_suffix(‘i386-unknown-linux2.6’, ‘.o’). library_prefix(‘i386-unknown-linux2.6’, ‘lib’). library_suffix(‘i386-unknown-linux2.6’, ‘.so’).

24. Test Specifications -24- Example Compiler % g++ specification paragraph comp_lang(‘g++’, ‘c++’). compiler_platform(‘g++’, Platform) :- platform(_, OS, _, Platform), OS \= ‘windows’. compiler_optimization_translate(‘g++’, ‘high’, ‘- O2’). comp_std_flags(‘g++’, ‘$(CXXFLAGS)’). comp_mutatee_flags(‘g++’, ‘$(MUTATEE_CXXFLAGS_GNU)’). mutatee_link_options(‘g++’, ‘$(MUTATEE_LDFLAGS_GNU)’).

25. Test Specifications -25- Status Infrastructure functional Converted 2/3 of old tests; working on the rest Will add SymtabAPI tests to new infrastructure Tech. report in early summer Future: integrate into NMI Build & Test system

26. Test Specifications -26- Pulled Slides

27. Test Specifications -27- Test Tuples Mutators mutator(‘example’, [‘example.C’]).  (example, [‘example.C’],, g++) Mutatees mutatee(‘example’, [‘example_mutatee.c’]). compiler_for_mutatee(‘example’, Compiler) :- compiler_language(Compiler, ‘c’); compiler_language(Compiler, ‘c++’).  (example, [‘example_mutatee.c’],, gcc, none)  (example, [‘example_mutatee.c’],, g++, high) ...

28. Test Specifications -28- Test Source Mutator mutator() { fn = findFunction(“f1”); p = findPoint(fn, ENTRY); fn2 = findFunction(“f2”); cl = funcCallExpr(fn2, NO_ARGS); insertSnippet(fn2, p); } Mutatee test_passes = FALSE; f2() { test_passes = TRUE; } f1() { if (test_passes) return PASSED; else return FAILED; }

29. Test Specifications -29- Example Test % Test specification paragraph for test_example test_name(‘test_example’). test_runs_everywhere(‘test_example’). mutator_mutatee(‘test_example’, ‘test_example’). mutator(‘test_example’, [‘test_example.C’]). mutatee(‘test_example’, [‘test_example_mutatee.c’]). compiler_for_mutatee(‘test_example’, Compiler) :- comp_lang(Compiler, ‘c’). test_runmode(‘test_example’, ‘both’). test_start_state(‘test_example’, ‘stopped’).

30. Test Specifications -30- Tuples Specification system outputs tuples for compilers, mutators, mutatees, and tests Build stage uses compiler, mutator, and mutatee tuples Run stage uses test tuples –(mutator, mutatee, run mode, start state)

31. Test Specifications -31- unit test 1 src unit test n src mutatee 1 src mutatee m src specspec compilerbuild scripts (makefiles) run scripts (test lists) mutatee compile stage (multiple compilers) mutator.so 1 mutator.so n mutatee 1_1 mutatee 1_k mutatee m_1 mutatee m_k test results 1_1test results m_1 database mutator compile stage test driver src Dyninst compiler test drivertest results 1_ktest results m_k

32. Test Specifications -32- Build Stage Specifications Run StageData Collection

33. Test Specifications -33- mutatee mutator Dyninst output test

34. Test Specifications -34- test driver mutatee mutator Dyninst output test

35. Test Specifications -35- Semi-Automated Test Generation Tools exist to automate unit test generation What about regression tests? Take a load off the programmer –Minimize direct interaction with test framework –Minimize details for test configuration –Automatically add test to makefiles, run scripts, and output parsing

36. Test Specifications -36- Compiler Specification mutator_compiler(‘g++’). mutatee_compiler(‘g++’). compiler_language(‘g++’, ‘c++’). compiler_optimization_translate(‘g++’, ‘none’, ‘-O0’). compiler_optimization_translate(‘g++’, ‘low’, ‘-O1’).... compiler_platform(‘g++’, Platform) :- platform(_, OS, _, Platform), OS \= ‘windows’. compiler_std_flags(‘g++’, ‘$(CXXFLAGS)’). compiler_mutatee_flags(‘g++’, ‘$(MUTATEE_CXXFLAGS_GNU)’). mutatee_link_options(‘g++’, ‘$(MUTATEE_LDFLAGS_GNU)’).