1. test automation for Fedora
AutoQA
test automation for Fedora
Presented by
Kamil Páral
Fedora QA team
License: CC-BY-SA

2. AutoQA recap
in a nutshell

3. AutoQA recap
There is a lot of stuff that Fedora QA team wants to test. Manual testing is slow and requires lots of hands. Let's automate some bits ⇝ AutoQA.
This way we can alleviate much work in release validation testing (Fedora Branched), proposed updates testing (proventesters) and other areas.

4. AutoQA architecture

5. AutoQA
now

6. AutoQA now
Bodhi “integration” (using comments)

7. AutoQA now
Bodhi “integration” (using comments)

8. AutoQA now Depcheck test
Will this update break dependencies in Fedora repositories?
Created by numerous hacks around YUM and RPM
Can't handle package conflicts and some other special cases => doesn't always work correctly

9. AutoQA now Upgradepath test
Will users of Fedora release N be able to update this package when upgrading to Fedora N+1?
Run only for stable Fedora repositories (not for updates-testing)
Intelligently handles updates proposed for multiple Fedora releases at once
Seems to be working really well (success++)

10. AutoQA now
The test report

11. AutoQA now We also have other tests
rpmlint, rpmguard, repoclosure, conflicts, rats_sanity, rats_install, anaconda_*, compose_tree, helloworld
Results not announced anywhere, just sent to our mailing list and to people who manually “opted-in”

12. AutoQA now

13. AutoQA
third-party tests

14. AutoQA 3rd-party tests
We used to encourage people to send us as much tests as they can create/find
We learned a lesson
We don't currently encourage people to supply more tests
Our energy is fully occupied with maintaining the current test suite and developing the framework
The framework is not yet ready for community- supplied tests

15. AutoQA 3rd-party tests Things we need to accomplish first:
storing community tests outside of our git
provide disposable test machines for (potentially) destructive tests
store and retrieve tests results easily – ResultsDB
relay user feedback to the test author, not us

16. AutoQA
near future?

17. AutoQA near future Staging server
hardware is ready, software support should be ready
should catch more errors before release
Continuous integration server
continuous execution of unit tests and integration tests
need to create mock objects of important third- party services (Koji, Bodhi)

18. AutoQA near future Maintenance-free test clients
current development version already support config-less clients, everything is installed automatically
creating disposable clients is harder – ideally should be integrated inside the provisioning tool (Autotest)
ResultsDB

19. ResultsDB

20. ResultsDB - what? stores AutoQA test results
should be able to store 'any' result
KeyVal pairs
tags (currently unused)
browse/use stored data

21. ResultsDB - why?
because mailing list is 'write-only' ;-)

22. ResultsDB - who? QA and RelEngs
"What can be safely pushed to stable / updates?"
Depcheck / Upgradepath
test plans / package acceptance criteria
generic test plan front-end
developers
"Stop the tons of spam!"
Bodhi integration, message bus (future tasks)

23. ResultsDB - who? anyone interested in 'metrics'
Pass / Fail ratio of a specific test in a Fedora release
failure rate of critpath packages
...

24. ResultsDB - current status
pre-alpha
deployed inside Red Hat VPN (non-public)
stable "Input API" (storing data)
patches for AutoQA awaiting review
XML-RPC interface

25. ResultsDB - near future
start gathering AutoQA test results
stabilize the Querying API
front-end to browse stored data (pschindl)
generic test plan front-end
provide metadata, get test plan matrix

26. ResultsDB - I want a pony!
Bodhi integration
Fedora Message Bus
asynchronous notifications
Rel-Eng tools

27. AutoQA
participation

28. AutoQA participation Useful links:
devel

29. AutoQA participation You can help with: framework development
quite complex, not suitable for a free afternoon
disposable test clients deployment
ideally integrated into Autotest
either creating on-demand or roll-backing (using LVM snapshots) to a predefined configuration
just virtual machines support for now

30. Questions? and maybe some answers kparal@redhat.com Contact:
License: CC-BY-SA

31. Attributions Impress template:
courtesy of Emily Dirsh (fedoraproject wiki)
Cliparts:
worktopussy: courtesy of Marcel 'ciscai' Trindade (flickr.com)