1. OpenStack QA Project- Roles & Its Key Activities
Ghanshyam Mann
Ken’ichi Ohmichi
NEC

2. Agenda Who are we ? Overview Of OpenStack Overview Of OpenStack QA
OpenStack QA Projects – Overview & Their Key Activities
Tempest
Tempest-lib
DevStack
Grenade
Hacking
Summary
Here is this presentation agenda.

3. Who are we ? Ghanshyam Mann Ken’ichi Ohmichi
Software developer from NEC
OpenStack community member since 2012
Core developer of Tempest and active contributor in Nova.
Ken’ichi Ohmichi
Core developer of Nova and Tempest

4. Overview of OpenStack

5. OpenStack Open source software for creating private and public clouds.
OpenStack software controls large pools of compute, storage,
and networking resources throughout a datacenter, managed
through a dashboard or via the OpenStack API.

6. OpenStack OpenStack Community has grown to lot bigger level.
Around 22K developers across over 160 countries, who have created around 3M LOC.
This is one of the fastest growing open source community.

7. OpenStack
OpenStack provides lot of services over cloud through many projects.
List Of OpenStack Projects-
Total 32 Projects and expected to grow.

8. Upstream Stability
Quality of OpenStack ?
OpenStack QA

9. OpenStack QA Project
A project that works on testing and quality assurance efforts across all OpenStack projects.
Main mission is to develop, maintain, and initiate tools and plans to ensure-
The upstream stability
Quality of OpenStack

10. OpenStack QA Project
The OpenStack QA project is made of multiple projects where each run independently towards the goals of the project.
Each project maintains a separate core team based on who provides reviews in those projects.
OpenStack QA
Tempest
OpenStack Integration Test Suite
Tempest-lib
OpenStack Functional Testing Library
DevStack
OpenStack development environment
Grenade
OpenStack Upgrade Testing
Hacking
A set of flake8 plugins that test and enforce the OpenStack Style Guidelines.

11. Tempest - OpenStack Integration Test Suite
This is a set of integration tests to be run against a live
OpenStack cluster.
Tempest is designed to be useful for a large number of different environments. This includes being useful for gating commits to OpenStack core projects, being used to validate OpenStack cloud implementations.

12. Tempest - OpenStack Integration Test Suite…
Type of tests in Tempest-
Tempest
API
Scenario
Stress
Third Party
Unit

13. Tempest - OpenStack Integration Test Suite…
API tests
API tests are validation tests for the OpenStack APIs.
Make sure stability and backward compatibility of APIs.
Scenario tests
Scenario tests are complex “through path” tests for OpenStack functionality.
Scenario tests interact with many services and tests cross projects features.
Stress tests
Stress tests are designed to stress an OpenStack environment by running a high workload against it and seeing what breaks.

14. Tempest - OpenStack Integration Test Suite…
Third Party API tests
Third party tests are tests for non native OpenStack APIs that are part of OpenStack projects.
An example is that Nova Compute currently has EC2 API support in tree.
Those should be kept separate from the normal OpenStack validation.
Unit tests 
Unit tests are the self checks for Tempest. 
They provide functional verification and regression checking for the internal components of Tempest.

15. Tempest - OpenStack Integration Test Suite…
Key Activities-
Consist of verity of tests like Scenario, Stress, API etc for lot of OpenStack Projects.
Thorough Cloud testing
Each Test Case can be uniquely identified with test UUID which can be used to execute or track a particular test case.
Uniquely identified Test case
User can select to execute tests sequentially or parallel.
Along with run time cred creation, parallel execution can be achieved with pre defined account in .yaml file
Sequentially & Parallel execution
Tempest clean up all resources created for tests automatically.
Automatic Cleanup
Tempest is Branchless which provide a facility to run it against all OpenStack stable branches.
Branchless

16. Tempest - OpenStack Integration Test Suite…
Focused Area-
Move common testing framework/modules to Tempest-lib.
Tempest Bug Triage and Fix
Since last year we started Bug Triage rotation for QA bugs.
This is helping us to keep new bug count less and provide
fast response to logged bugs.
Recently in OpenStack summit, we discussed about providing External plugin interface.

17. Tempest-lib - OpenStack Functional Testing Library
Motivation Behind Tempest-lib

18. Tempest-lib - OpenStack Functional Testing Library
As number of OpenStack projects keep growing, some issues comes up regarding their testing -
It becomes difficult to tests all projects in Tempest.
There will be duplicate effort for developing functional tests in each Projects.
To solve such issues, its better to provide common testing framework through some library so that projects can develop their own tests by using common framework.

19. Tempest-lib - OpenStack Functional Testing Library
A library to assist in creating functional or integrated test suites for OpenStack projects.
Tempest-lib is a library of common functionality that was originally in Tempest.
Common Testing
Framework
Tempest
Tempest-lib

20. Tempest-lib - OpenStack Functional Testing Library…
Key Activities-
OpenStack Projects can develop their functional test using Tempest-lib common testing framework.
Provide common testing framework
OpenStack python-* client can develop their functional tests using common CLI testing framework.
Provide common CLI testing framework
A output filter for subunit streams. Useful in conjunction with calling a test runner that emits subunit.
subunit-trace
A unified REST Client which provides base layer for wrapping outgoing http request and provide response code checking and error handling,
REST Client
Auth Provider which provide authentication based on Keystone V2 & V3 API.
Auth
skip_because: Skip a test because of a bug
find_test_caller: Perform stack introspection to find the test caller.
Utility functions

21. Tempest-lib - OpenStack Functional Testing Library…
Project Functional tests can be developed using Tempest-lib.
Tempest tests started using Tempest-lib.
Tempest tests
Tempest-lib
Project-n
Functional
tests
Project-1

22. DevStack - OpenStack development environment
DevStack is an opinionated script to quickly create an OpenStack development environment.
It can also be used to demonstrate starting/running OpenStack services and provide examples of using them from a command line.
Work on OpenStack
Developer
Install OpenStack Development Environment
DevStack

23. DevStack - OpenStack development environment…
 Key Activities-
Provide easier environment for developers to dive into OpenStack so that they can productively contribute without having to understand every part of the system at once.
Provide OpenStack Development Environment
Provide an environment for the OpenStack CI testing on every commit to the projects
Provide Environment for CI testing
Provide easy way to prototype cross-project features.
Way to prototype cross-project features
Along with Extras.d hooks, DevStack supports a standard mechanism for including plugins from external repositories.
Externally Hosted Plugins
The default services configured by DevStack are Identity (keystone), Object Storage (swift), Image Service (glance), Block Storage (cinder), Compute (nova), Networking (nova), Dashboard (horizon), heat.
Additional services can be included in DevStack.
Services

24. DevStack - OpenStack development environment…
 Supported Components -
Ubuntu
Fedora
RHEL
Base OS
MySQL,
PostgreSQL
Databases
Rabbit
Qpid
Queues
Nova Network
Neutron
OpenStack Network
single node
multi-node – not much tested
Node Configurations

25. DevStack - OpenStack development environment…
Focused Area -
Making Nova v2.1 API as default
Nova has two set of APIs, v2 and v2.1. Currently DevStack configure v2 API as default and provide option to configure v2.1 API on /v2 endpoint in targeted environment. As v2 API will be deprecated in future, it will be better to make v2.1 as default Nova API in DevStack to tests v2.1 API thoroughly.
Make external plugins for projects currently having their scripts in DevStack tree.

26. Grenade – OpenStack Upgrade Testing
Grenade is an OpenStack test harness to exercise the upgrade process between releases.
It uses DevStack to perform an initial OpenStack install and as a reference for the final configuration.
Currently Grenade can upgrade Keystone, Glance, Nova, Neutron, Cinder, Swift, Ceilometer, Horizon and Ironic in their default DevStack configurations. 

27. Grenade – OpenStack Upgrade Testing…
How Grenade ensure that any Code modification does not break Upgrade.
Upgrade From
KILO to
Master
Grenade
OpenStack
KILO
Tests changes for
upgrade through
grenade gate job
PASS
Merge
Target Branch
(Master) 1 2 3 4
OpenStack
Master Branch
FAIL
Do Not Merge
Code modification
Developer

28. Grenade – OpenStack Upgrade Testing…
Key Activities-
New code should work with old configs
New code should need nothing more than 'db migrations'
Resources created by services before upgrade, should still be there after the system is upgraded
Any other required changes on upgrade are an exception and must be called out in the release notes.
Theory of Upgrade
Block unintentional project changes that would break the Theory of Upgrade.
Thorough upgrade testing
Ensure that upgrading a cloud doesn't do something dumb like delete and recreate all your servers/volumes/networks.
Validate successful upgrade
Grenade jobs are now running on every patch in gate for projects that support upgrade.
Upgrade testing in gate

29. Hacking
Hacking is a set of flake8 plugins that test and enforce the OpenStack Style Guidelines.
Most of the additional style guidelines that OpenStack has taken on came from the Google Python Style Guide.
Since then, a few more OpenStack specific ones have been added or modified.
OpenStack Style Guidelines
OpenStack has a set of style guidelines for clarity. OpenStack is a very large code base (over 1 Million lines of python), spanning dozens of git trees, with over a thousand developers contributing every 12 months. As such common style helps developers understand code in reviews, move between projects smoothly, and overall make the code more maintainable.
Example-
[H201] Do not write except:, use except Exception: at the very least. When catching an exception you should be as specific so you don’t mistakenly catch unexpected exceptions.
[H301] Do not import more than one module per line (*)
[H102 H103] Newly contributed Source Code should be licensed under the Apache 2.0 license. All source files should have the following header:
Commit message
Using a common format for commit messages will help keep our git history readable.
For further information on constructing high quality commit messages, and how to split up commits into a series of changes, consult the project wiki:

30. Conclusion & Key Points
OpenStack QA helps to validate your OpenStack Cloud.
Migrate more testing common code from Tempest to Tempest-lib.
Tempest-lib to avoid duplicate test development effort.
External Plugins for DevStack, Tempest, Grenade.
More Info -
IRC - #openstack-qa