OpenStack QA Project- Roles & Its Key Activities Ghanshyam Mann Ken’ichi Ohmichi NEC

Agenda Who are we ? Overview Of OpenStack Overview Of OpenStack QA OpenStack QA Projects – Overview & Their Key Activities Tempest Tempest-lib DevStack Grenade Hacking Summary Here is this presentation agenda.

Who are we ? Ghanshyam Mann Ken’ichi Ohmichi Software developer from NEC OpenStack community member since 2012 Core developer of Tempest and active contributor in Nova. Ken’ichi Ohmichi Core developer of Nova and Tempest

Overview of OpenStack

OpenStack Open source software for creating private and public clouds. OpenStack software controls large pools of compute, storage, and networking resources throughout a datacenter, managed through a dashboard or via the OpenStack API.

OpenStack OpenStack Community has grown to lot bigger level. Around 22K developers across over 160 countries, who have created around 3M LOC. This is one of the fastest growing open source community.

OpenStack OpenStack provides lot of services over cloud through many projects. List Of OpenStack Projects- Total 32 Projects and expected to grow.

Upstream Stability Quality of OpenStack ? OpenStack QA

OpenStack QA Project A project that works on testing and quality assurance efforts across all OpenStack projects. Main mission is to develop, maintain, and initiate tools and plans to ensure- The upstream stability Quality of OpenStack

OpenStack QA Project The OpenStack QA project is made of multiple projects where each run independently towards the goals of the project. Each project maintains a separate core team based on who provides reviews in those projects. OpenStack QA Tempest OpenStack Integration Test Suite Tempest-lib OpenStack Functional Testing Library DevStack OpenStack development environment Grenade OpenStack Upgrade Testing Hacking A set of flake8 plugins that test and enforce the OpenStack Style Guidelines.

Tempest - OpenStack Integration Test Suite This is a set of integration tests to be run against a live OpenStack cluster. Tempest is designed to be useful for a large number of different environments. This includes being useful for gating commits to OpenStack core projects, being used to validate OpenStack cloud implementations.

Tempest - OpenStack Integration Test Suite… Type of tests in Tempest- Tempest API Scenario Stress Third Party Unit

Tempest - OpenStack Integration Test Suite… API tests API tests are validation tests for the OpenStack APIs. Make sure stability and backward compatibility of APIs. Scenario tests Scenario tests are complex “through path” tests for OpenStack functionality. Scenario tests interact with many services and tests cross projects features. Stress tests Stress tests are designed to stress an OpenStack environment by running a high workload against it and seeing what breaks.

Tempest - OpenStack Integration Test Suite… Third Party API tests Third party tests are tests for non native OpenStack APIs that are part of OpenStack projects. An example is that Nova Compute currently has EC2 API support in tree. Those should be kept separate from the normal OpenStack validation. Unit tests  Unit tests are the self checks for Tempest.  They provide functional verification and regression checking for the internal components of Tempest.

Tempest - OpenStack Integration Test Suite… Key Activities- Consist of verity of tests like Scenario, Stress, API etc for lot of OpenStack Projects. Thorough Cloud testing Each Test Case can be uniquely identified with test UUID which can be used to execute or track a particular test case. Uniquely identified Test case User can select to execute tests sequentially or parallel. Along with run time cred creation, parallel execution can be achieved with pre defined account in .yaml file Sequentially & Parallel execution Tempest clean up all resources created for tests automatically. Automatic Cleanup Tempest is Branchless which provide a facility to run it against all OpenStack stable branches. Branchless

Tempest - OpenStack Integration Test Suite… Focused Area- Move common testing framework/modules to Tempest-lib. Tempest Bug Triage and Fix Since last year we started Bug Triage rotation for QA bugs. This is helping us to keep new bug count less and provide fast response to logged bugs. Recently in OpenStack summit, we discussed about providing External plugin interface.

Tempest-lib - OpenStack Functional Testing Library Motivation Behind Tempest-lib

Tempest-lib - OpenStack Functional Testing Library As number of OpenStack projects keep growing, some issues comes up regarding their testing - It becomes difficult to tests all projects in Tempest. There will be duplicate effort for developing functional tests in each Projects. To solve such issues, its better to provide common testing framework through some library so that projects can develop their own tests by using common framework.

Tempest-lib - OpenStack Functional Testing Library A library to assist in creating functional or integrated test suites for OpenStack projects. Tempest-lib is a library of common functionality that was originally in Tempest. Common Testing Framework Tempest Tempest-lib

Tempest-lib - OpenStack Functional Testing Library… Key Activities- OpenStack Projects can develop their functional test using Tempest-lib common testing framework. Provide common testing framework OpenStack python-* client can develop their functional tests using common CLI testing framework. Provide common CLI testing framework A output filter for subunit streams. Useful in conjunction with calling a test runner that emits subunit. subunit-trace A unified REST Client which provides base layer for wrapping outgoing http request and provide response code checking and error handling, REST Client Auth Provider which provide authentication based on Keystone V2 & V3 API. Auth skip_because: Skip a test because of a bug find_test_caller: Perform stack introspection to find the test caller. Utility functions

Tempest-lib - OpenStack Functional Testing Library… Project Functional tests can be developed using Tempest-lib. Tempest tests started using Tempest-lib. Tempest tests Tempest-lib Project-n Functional tests Project-1

DevStack - OpenStack development environment DevStack is an opinionated script to quickly create an OpenStack development environment. It can also be used to demonstrate starting/running OpenStack services and provide examples of using them from a command line. Work on OpenStack Developer Install OpenStack Development Environment DevStack

DevStack - OpenStack development environment…  Key Activities- Provide easier environment for developers to dive into OpenStack so that they can productively contribute without having to understand every part of the system at once. Provide OpenStack Development Environment Provide an environment for the OpenStack CI testing on every commit to the projects Provide Environment for CI testing Provide easy way to prototype cross-project features. Way to prototype cross-project features Along with Extras.d hooks, DevStack supports a standard mechanism for including plugins from external repositories. Externally Hosted Plugins The default services configured by DevStack are Identity (keystone), Object Storage (swift), Image Service (glance), Block Storage (cinder), Compute (nova), Networking (nova), Dashboard (horizon), heat. Additional services can be included in DevStack. Services

DevStack - OpenStack development environment…  Supported Components - Ubuntu Fedora RHEL Base OS MySQL, PostgreSQL Databases Rabbit Qpid Queues Nova Network Neutron OpenStack Network single node multi-node – not much tested Node Configurations

DevStack - OpenStack development environment… Focused Area - Making Nova v2.1 API as default Nova has two set of APIs, v2 and v2.1. Currently DevStack configure v2 API as default and provide option to configure v2.1 API on /v2 endpoint in targeted environment. As v2 API will be deprecated in future, it will be better to make v2.1 as default Nova API in DevStack to tests v2.1 API thoroughly. Make external plugins for projects currently having their scripts in DevStack tree.

Grenade – OpenStack Upgrade Testing Grenade is an OpenStack test harness to exercise the upgrade process between releases. It uses DevStack to perform an initial OpenStack install and as a reference for the final configuration. Currently Grenade can upgrade Keystone, Glance, Nova, Neutron, Cinder, Swift, Ceilometer, Horizon and Ironic in their default DevStack configurations. 

Grenade – OpenStack Upgrade Testing… How Grenade ensure that any Code modification does not break Upgrade. Upgrade From KILO to Master Grenade OpenStack KILO Tests changes for upgrade through grenade gate job PASS Merge Target Branch (Master) 1 2 3 4 OpenStack Master Branch FAIL Do Not Merge Code modification Developer

Grenade – OpenStack Upgrade Testing… Key Activities- New code should work with old configs New code should need nothing more than 'db migrations' Resources created by services before upgrade, should still be there after the system is upgraded Any other required changes on upgrade are an exception and must be called out in the release notes. Theory of Upgrade Block unintentional project changes that would break the Theory of Upgrade. Thorough upgrade testing Ensure that upgrading a cloud doesn't do something dumb like delete and recreate all your servers/volumes/networks. Validate successful upgrade Grenade jobs are now running on every patch in gate for projects that support upgrade. Upgrade testing in gate

Hacking Hacking is a set of flake8 plugins that test and enforce the OpenStack Style Guidelines. Most of the additional style guidelines that OpenStack has taken on came from the Google Python Style Guide. http://google-styleguide.googlecode.com/svn/trunk/pyguide.html Since then, a few more OpenStack specific ones have been added or modified. OpenStack Style Guidelines OpenStack has a set of style guidelines for clarity. OpenStack is a very large code base (over 1 Million lines of python), spanning dozens of git trees, with over a thousand developers contributing every 12 months. As such common style helps developers understand code in reviews, move between projects smoothly, and overall make the code more maintainable. Example- [H201] Do not write except:, use except Exception: at the very least. When catching an exception you should be as specific so you don’t mistakenly catch unexpected exceptions. [H301] Do not import more than one module per line (*) [H102 H103] Newly contributed Source Code should be licensed under the Apache 2.0 license. All source files should have the following header: Commit message Using a common format for commit messages will help keep our git history readable. For further information on constructing high quality commit messages, and how to split up commits into a series of changes, consult the project wiki: https://wiki.openstack.org/GitCommitMessages

Conclusion & Key Points OpenStack QA helps to validate your OpenStack Cloud. Migrate more testing common code from Tempest to Tempest-lib. Tempest-lib to avoid duplicate test development effort. External Plugins for DevStack, Tempest, Grenade. More Info - https://wiki.openstack.org/wiki/QA https://wiki.openstack.org/wiki/Meetings/QATeamMeeting IRC - #openstack-qa