test automation for Fedora AutoQA test automation for Fedora Presented by Kamil Páral Fedora QA team License: CC-BY-SA

AutoQA recap in a nutshell

AutoQA recap There is a lot of stuff that Fedora QA team wants to test. Manual testing is slow and requires lots of hands. Let's automate some bits ⇝ AutoQA. This way we can alleviate much work in release validation testing (Fedora Branched), proposed updates testing (proventesters) and other areas.

AutoQA architecture

AutoQA now

AutoQA now Bodhi “integration” (using comments)

AutoQA now Bodhi “integration” (using comments)

AutoQA now Depcheck test Will this update break dependencies in Fedora repositories? Created by numerous hacks around YUM and RPM Can't handle package conflicts and some other special cases => doesn't always work correctly

AutoQA now Upgradepath test Will users of Fedora release N be able to update this package when upgrading to Fedora N+1? Run only for stable Fedora repositories (not for updates-testing) Intelligently handles updates proposed for multiple Fedora releases at once Seems to be working really well (success++)

AutoQA now The test report

AutoQA now We also have other tests rpmlint, rpmguard, repoclosure, conflicts, rats_sanity, rats_install, anaconda_*, compose_tree, helloworld Results not announced anywhere, just sent to our mailing list and to people who manually “opted-in”

AutoQA now

AutoQA third-party tests

AutoQA 3rd-party tests We used to encourage people to send us as much tests as they can create/find We learned a lesson We don't currently encourage people to supply more tests Our energy is fully occupied with maintaining the current test suite and developing the framework The framework is not yet ready for community- supplied tests

AutoQA 3rd-party tests Things we need to accomplish first: storing community tests outside of our git provide disposable test machines for (potentially) destructive tests store and retrieve tests results easily – ResultsDB relay user feedback to the test author, not us

AutoQA near future?

AutoQA near future Staging server hardware is ready, software support should be ready should catch more errors before release Continuous integration server continuous execution of unit tests and integration tests need to create mock objects of important third- party services (Koji, Bodhi)

AutoQA near future Maintenance-free test clients current development version already support config-less clients, everything is installed automatically creating disposable clients is harder – ideally should be integrated inside the provisioning tool (Autotest) ResultsDB

ResultsDB

ResultsDB - what? stores AutoQA test results should be able to store 'any' result KeyVal pairs tags (currently unused) browse/use stored data

ResultsDB - why? because mailing list is 'write-only' ;-)

ResultsDB - who? QA and RelEngs "What can be safely pushed to stable / updates?" Depcheck / Upgradepath test plans / package acceptance criteria generic test plan front-end developers "Stop the tons of email spam!" Bodhi integration, message bus (future tasks)

ResultsDB - who? anyone interested in 'metrics' Pass / Fail ratio of a specific test in a Fedora release failure rate of critpath packages ...

ResultsDB - current status pre-alpha deployed inside Red Hat VPN (non-public) stable "Input API" (storing data) patches for AutoQA awaiting review XML-RPC interface

ResultsDB - near future start gathering AutoQA test results stabilize the Querying API front-end to browse stored data (pschindl) generic test plan front-end provide metadata, get test plan matrix

ResultsDB - I want a pony! Bodhi integration Fedora Message Bus asynchronous notifications Rel-Eng tools

AutoQA participation

AutoQA participation Useful links: https://fedoraproject.org/wiki/AutoQA https://fedorahosted.org/autoqa https://fedorahosted.org/mailman/listinfo/autoqa- devel

AutoQA participation You can help with: framework development quite complex, not suitable for a free afternoon disposable test clients deployment ideally integrated into Autotest either creating on-demand or roll-backing (using LVM snapshots) to a predefined configuration just virtual machines support for now

Questions? and maybe some answers kparal@redhat.com Contact: License: CC-BY-SA

Attributions Impress template: courtesy of Emily Dirsh (fedoraproject wiki) Cliparts: worktopussy: courtesy of Marcel 'ciscai' Trindade (flickr.com)