1. AutoQA and You Will Woods, Fedora QA Architect FUDCon Toronto 2009

2. Overview / Hot Dog ● History of AutoQA ● Current State ● Future Plans ● Getting Involved / Contributing Tests ● Resources

3. AutoQA History

4. Big Problems ● Broken Rawhide ● Broken Repos ● Broken Deps ● Broken Hearts

5. Poor ESR “After thirteen years as a loyal Red Hat and Fedora user, I reached my limit today, when an attempt to upgrade one (1) package pitched me into a four-hour marathon of dependency chasing, at the end of which an attempt to get around a trivial file conflict rendered my system unusable.” Eric S. Raymond

6. Big Solutions? ● Beaker (testing.108.redhat.com)

7. Breaking It Down ● FUDCon “What is Rawhide” discussion – What makes Rawhide “broken” or “good”? – Can we write code to check that? – How can we run that code automatically? ● FUDCon AutoQA discussion/hackfest – “When stuff happens, run tests” ● FAD: Is Rawhide Broken proposal – Rawhide Acceptance Test Plan

8. JFDI ● FUDCon hackfest for autoqa 0.1 – Event watcher, test harness, and one test ● Adopted autotest for job scheduling etc. – Nobody really wants to write their own test harness / test machine provisioning system / job scheduler / etc. ● Rewrote autoqa in Python ● Standardized event hooks and tests

9. AutoQA: Current State

10. Overall Design ● Event-driven automated testing – “When something happens, run some tests” ● Well-defined event hooks – Define the event/trigger – Define test inputs ● Tests are independent – Can run outside of the test system

11. Current Implementation ● Event watcher for each hook ● 'autoqa' harness script ● Per-test control file – Required to queue a new autotest job ● Autotest picks a test host and runs test according to control file instructions ● Test can report results however it likes – Autotest saves output/results

13. Current Hooks ● post-repo-update – Runs when repo metadata changes ● post-tree-compose – Runs when new installer images appear ● post-koji-build – Runs when a package build finishes

14. Current tests ● post-repo-update – conflicts, repoclosure, rats_sanity ● post-tree-compose – rats_install ● post-koji-build – rpmlint, rpmguard (in progress)

15. AutoQA Python Library ● Package of useful Python modules for writing watchers and tests. – util, config, rats, virtguest, repoinfo, koji_utils

16. israwhidebroken.com ● Test result database ● TurboGears frontend ● Store and display results for the 14 test cases defined by RATS ● Answers the question “is Rawhide broken?” ● We can use these results to select the Last Known Good Rawhide images

18. Current Setup ● Internal pilot instance – Test server, 5 test hosts – Israwhidebroken running on test server – Simple test results emailed to autoqa-results mailing list – Full test results saved on autotest server

19. Future Plans

20. Coming Soon ● Public Autotest instance – FAS authentication required ● post-bodhi-update hook – Run tests when a new update is added or requests a push ● AutoQA Depcheck Test – No more broken deps in public repos!

21. Coming Soon..ish ● Easy AutoQA for packagers – Add post-koji-build or post-bodhi-update tests right in pkgcvs - “make new-test” ● post-iso-build hook – Lets us start to automate our huge install test plan ● Use Fedora Message Bus, not watchers

22. Coming Eventually Someday Probably Maybe ● Multi-host / networking tests – httpd, nfs, etc. ● Functional testing for GUIs – But how do we script that? ● World Domination

23. Getting Involved

24. How To Contribute Tests ● WRITE THE TEST FIRST. – Don't worry about AutoQA yet. ● Write a test that you can run on its own –./radtest ARG1 ARG2... ● Until you are satisfied with the way the test works you don't need to know anything about AutoQA 1,2

25. 1 Except perhaps for some helpful test assumptions ● Tests will have a resultsdir and a tempdir ● Tests will have high-speed network access to all Fedora repos ● Koji, Bodhi, etc. will be reachable ● There isn't any state stored between tests

26. 2 And some details about the current hooks ● post-tree-compose args: – Tree name, URL, arches ● post-repo-update args: – Repo name, URL, arches, parent repo names ● post-koji-build args: – Package name, NVR, koji tag, arches

27. References ● http://fedoraproject.org/wiki/AutoQA http://fedoraproject.org/wiki/AutoQA ● http://fedoraproject.org/wiki/AutoQA_architecture http://fedoraproject.org/wiki/AutoQA_architecture ● http://fedoraproject.org/wiki/Writing_AutoQA_Tests http://fedoraproject.org/wiki/Writing_AutoQA_Tests ● http://fedoraproject.org/wiki/Writing_AutoQA_Hooks http://fedoraproject.org/wiki/Writing_AutoQA_Hooks ● https://fedorahosted.org/autoqa/ https://fedorahosted.org/autoqa/ ● http://autotest.kernel.org/ http://autotest.kernel.org/ ● http://wwoods.fedorapeople.org/hot-dog/ http://wwoods.fedorapeople.org/hot-dog/

28. Questions?