AutoQA and You Will Woods, Fedora QA Architect FUDCon Toronto 2009

Overview / Hot Dog ● History of AutoQA ● Current State ● Future Plans ● Getting Involved / Contributing Tests ● Resources

AutoQA History

Big Problems ● Broken Rawhide ● Broken Repos ● Broken Deps ● Broken Hearts

Poor ESR “After thirteen years as a loyal Red Hat and Fedora user, I reached my limit today, when an attempt to upgrade one (1) package pitched me into a four-hour marathon of dependency chasing, at the end of which an attempt to get around a trivial file conflict rendered my system unusable.” Eric S. Raymond

Big Solutions? ● Beaker (testing.108.redhat.com)

Breaking It Down ● FUDCon “What is Rawhide” discussion – What makes Rawhide “broken” or “good”? – Can we write code to check that? – How can we run that code automatically? ● FUDCon AutoQA discussion/hackfest – “When stuff happens, run tests” ● FAD: Is Rawhide Broken proposal – Rawhide Acceptance Test Plan

JFDI ● FUDCon hackfest for autoqa 0.1 – Event watcher, test harness, and one test ● Adopted autotest for job scheduling etc. – Nobody really wants to write their own test harness / test machine provisioning system / job scheduler / etc. ● Rewrote autoqa in Python ● Standardized event hooks and tests

AutoQA: Current State

Overall Design ● Event-driven automated testing – “When something happens, run some tests” ● Well-defined event hooks – Define the event/trigger – Define test inputs ● Tests are independent – Can run outside of the test system

Current Implementation ● Event watcher for each hook ● 'autoqa' harness script ● Per-test control file – Required to queue a new autotest job ● Autotest picks a test host and runs test according to control file instructions ● Test can report results however it likes – Autotest saves output/results

Current Hooks ● post-repo-update – Runs when repo metadata changes ● post-tree-compose – Runs when new installer images appear ● post-koji-build – Runs when a package build finishes

Current tests ● post-repo-update – conflicts, repoclosure, rats_sanity ● post-tree-compose – rats_install ● post-koji-build – rpmlint, rpmguard (in progress)

AutoQA Python Library ● Package of useful Python modules for writing watchers and tests. – util, config, rats, virtguest, repoinfo, koji_utils

israwhidebroken.com ● Test result database ● TurboGears frontend ● Store and display results for the 14 test cases defined by RATS ● Answers the question “is Rawhide broken?” ● We can use these results to select the Last Known Good Rawhide images

Current Setup ● Internal pilot instance – Test server, 5 test hosts – Israwhidebroken running on test server – Simple test results ed to autoqa-results mailing list – Full test results saved on autotest server

Future Plans

Coming Soon ● Public Autotest instance – FAS authentication required ● post-bodhi-update hook – Run tests when a new update is added or requests a push ● AutoQA Depcheck Test – No more broken deps in public repos!

Coming Soon..ish ● Easy AutoQA for packagers – Add post-koji-build or post-bodhi-update tests right in pkgcvs - “make new-test” ● post-iso-build hook – Lets us start to automate our huge install test plan ● Use Fedora Message Bus, not watchers

Coming Eventually Someday Probably Maybe ● Multi-host / networking tests – httpd, nfs, etc. ● Functional testing for GUIs – But how do we script that? ● World Domination

Getting Involved

How To Contribute Tests ● WRITE THE TEST FIRST. – Don't worry about AutoQA yet. ● Write a test that you can run on its own –./radtest ARG1 ARG2... ● Until you are satisfied with the way the test works you don't need to know anything about AutoQA 1,2

1 Except perhaps for some helpful test assumptions ● Tests will have a resultsdir and a tempdir ● Tests will have high-speed network access to all Fedora repos ● Koji, Bodhi, etc. will be reachable ● There isn't any state stored between tests

2 And some details about the current hooks ● post-tree-compose args: – Tree name, URL, arches ● post-repo-update args: – Repo name, URL, arches, parent repo names ● post-koji-build args: – Package name, NVR, koji tag, arches

References ● ● ● ● ● ● ●

Questions?