1. Created by Jan Medved www.opendaylight.org Integration & Test Strategy for Lithium

2. Created by Jan Medved www.opendaylight.org  OpenDaylight is made of multiple projects, each project defines one or more features -> we have to test them individually and together.  Projects are quite heterogeneous: platform, tools, plugins, apps, etc… -> they have different test needs.  OpenDaylight has a common CI process for all projects -> tests should in general be part of the CI.  We leave users freedom to select features -> we need an strategy to test a non-rigid distribution. Facts 2

3. Created by Jan Medved www.opendaylight.org  Instead of verifying projects, test user-facing features.  Test single feature enabled to identify issues in a feature itself.  Test all features enabled to identify feature interferences.  For a given project, test after:  A change in the project itself. Today we miss system test on code submit, but this is also possible.  A change (merge) in an upstream project. This will trigger build and system test.  Be selective with the tests we run:  A change in a “stem” project will trigger test on many features.  A change in a “leaf” project will trigger test on few concerned features. High Level Strategy 3

4. Created by Jan Medved www.opendaylight.org We define 2 integration features for test purposes:  compatible-with-all -> useful for all features system test  all = compatible-with-all + incompatible features -> useful to detect feature/bundle interferences We generate 2 distributions (no feature enabled at startup):  Regular distribution with all projects feature indexes (except integration)  Test distribution = Regular distribution + integration feature index Distribution Assembly 4

5. Created by Jan Medved www.opendaylight.org CI Infrastructure 5 System Test (Robot FW) System Test (Robot FW) Post test results Run test suite Project Build and Test (Jenkins) Project Build and Test (Jenkins) Code management (GIT/Gerrit) Code management (GIT/Gerrit) Gerrit code submit or merge event Artifact Repository (Nexus) Artifact Repository (Nexus) Upload artifactsDownload artifacts Project Build and Test (Jenkins) Project Build and Test (Jenkins) Upstream project merge

6. Created by Jan Medved www.opendaylight.org Build test occurs during project code build. Examples are Junit, PAX-EXAM or Karaf feature install test Every project in OpenDaylight defines 3 build jobs:  Verify -> This job builds the project code and pass build tests upon code submit  Merge -> This job builds the project code and pass build tests upon code merge. In addition it will upload generated artifacts to Nexus repository.  Integration -> This job will build the project code and pass build tests after a merge in an upstream project Build Test 6

7. Created by Jan Medved www.opendaylight.org System test requires a framework that installs controller and executes tests against network tools. Example is the Robot test. For every project user-facing feature we will create 2 system test jobs:  feature-only test -> Install single feature with required dependencies and run feature planned test  feature-all test  If feature is compatible -> Install odl-integration-compatible- with-all and run feature planned test  If feature is not compatible -> Install odl-integration- compatible-with-all + incompatible feature and run feature planned test System Test 7

8. Created by Jan Medved www.opendaylight.org Test Triggers 8 Code Merge Release creation Merge build feature-only test feature-all test feature-only test feature-all test For all available features For all features in downstream Code Submit Verify build *Optional (requires project distribution or some automation to update project artifact) Integration build For all projects in downstream feature-only test*

9. Created by Jan Medved www.opendaylight.org  Code sanity and code coverage reports are collected daily to Sonar  All system test code will be stored in the Integration repo, we can quantify the amount of test per project  Maven plugin for API coverage?  Number of critical bugs per project and bugs trends Quality metrics 9