Business Unit or Product Name © 2007 IBM Corporation Introduction of Autotest Qing Lin

Business Unit or Product Name © 2007 IBM Corporation 2Jun 19, 2012 Agenda  Overview of Autotest  Autotest Structure  Autotest Client Harness  Adding tests to Autotest Client  Analyze the test results  Integrate in Jenkins

Business Unit or Product Name © 2007 IBM Corporation 3Jun 19, 2012 Overview of Autotest – What is Autotest?  Autotest is a framework for fully automated testing.  It is designed primarily to test the Linux kernel, though it is useful for many other purposes such as qualifying new hardware, virtualization testing and other general user space program testing under linux platforms.  It's an open-source project under the GPL and is used and developed by a number of organizations, including Google, IBM, Red Hat, and many others.

Business Unit or Product Name © 2007 IBM Corporation 4Jun 19, 2012 Overview of Autotest – Goal  The aim is to produce a client which is open source, implicitly handles errors, produces consistent results, is easily installable, simple to maintain and runs either standalone or within any server harness.

Business Unit or Product Name © 2007 IBM Corporation 5Jun 19, 2012 Overview of Autotest – General Features  Support for functional, performance, stress tests  Simple inclusion of tests – Trivial to incorporate existing tests – Easy to write new tests – Possible to write very complex tests  Complete control over environment  Consistent, machine-parseable results output  Tests can be executed standalone or as part of a larger-scale system – Easy to run a prepackaged collection of tests

Business Unit or Product Name © 2007 IBM Corporation 6Jun 19, 2012 Autotest Structure  Users-->Autotest Client  Users-->Autoserver-->Autotest Client  Users-->Frontend-->Autoserver-->Autotest Client

Business Unit or Product Name © 2007 IBM Corporation 7Jun 19, 2012 Autotest Client Harness  Harness running on the machine under test  Written in Python– APIs to easily compile and run C code  Each test has a simple wrapper class  Standardized results reporting for all tests  Comprehensive error handling  Control flow is managed by a control file

Business Unit or Product Name © 2007 IBM Corporation 8Jun 19, 2012 Autotest Client Harness – Anatomy of a test

Business Unit or Product Name © 2007 IBM Corporation 9Jun 19, 2012 Autotest Client Harness – Test Class  initialize() - This is run before everything, every time the test is run.  setup() - This is run when you first use the test, and normally is used to compile the source code.  warmup() -For performance tests that need to conduct any pre-test priming to make the results valid. This is called by job.run_test before running the test itself, but after all the setup.  run_once() - This is called by job.run_test N times, where N is controlled by the iterations parameter to run_test (defaulting to one). It also gets called an additional time with profilers turned on, if you have any profilers enabled.  postprocess_iteration() - This processes any results generated by the test iteration, and writes them out into a keyval. It's generally not called for the profiling iteration, as that may have different performance.  postprocess() - [DEPRECATED] This is called once to do postprocessing of test iterations, after all iterations are complete. Please use postprocess_iteration instead.  cleanup() -Used for any post-test cleanup. It always gets called, regardless of the success or failure of the test execution.

Business Unit or Product Name © 2007 IBM Corporation 10Jun 19, 2012 Adding tests to Autotest Client – Contents Today  Tests / Testsuites  It only takes about 10 minutes to wrap a test

Business Unit or Product Name © 2007 IBM Corporation 11Jun 19, 2012 Adding tests to Autotest Client  An example control file (eg tests/mytest/control)  A test wrapper (eg (tests/mytest/mytest.py)  Some source code for the test (if it's not all done in just the python script)  The name of the subdirectory (tests/mytest), the test wrapper (mytest.py) and the name of the class inside the test wrapper (mytest), all match.

Business Unit or Product Name © 2007 IBM Corporation 12Jun 19, 2012 Adding tests to Autotest Client – control file  Path :tests/mytest/control

Business Unit or Product Name © 2007 IBM Corporation 13Jun 19, 2012 Adding tests to Autotest Client – test wrapper  Path :tests/mytest/mytest.py

Business Unit or Product Name © 2007 IBM Corporation 14Jun 19, 2012 Adding tests to Autotest Client – test call process

Business Unit or Product Name © 2007 IBM Corporation 15Jun 19, 2012 Analyze the test result  View the rough result –./tools/scan_results.py results/default/status

Business Unit or Product Name © 2007 IBM Corporation 16Jun 19, 2012 Analyze the test result  Result directory – results/default  Detailed info for each case

Business Unit or Product Name © 2007 IBM Corporation 17Jun 19, 2012 Analyze the test result  Result in HTML

Business Unit or Product Name © 2007 IBM Corporation 18Jun 19, 2012 Integrate into Jenkins – Create a Job

Business Unit or Product Name © 2007 IBM Corporation 19Jun 19, 2012 Integrate into Jenkins – Configuration I Name Workspace Configuration Page

Business Unit or Product Name © 2007 IBM Corporation 20Jun 19, 2012 Integrate into Jenkins – Configuration II Run test script

Business Unit or Product Name © 2007 IBM Corporation 21Jun 19, 2012 Integrate into Jenkins – Configuration III Notification

Business Unit or Product Name © 2007 IBM Corporation 22Jun 19, 2012 Integrate into Jenkins – Get the Result in Log

Business Unit or Product Name © 2007 IBM Corporation 23Jun 19, 2012 Thank you Thanks to Li and Richard for reviewing it.