Overview on CI Use JJB (Jenkins Job Builder) to manage Jenkins jobs. Supports x86_64, s390x build platforms. Packer image to cache the build artifacts and load them in x86_64 build nodes. https://github.com/hyperledger/ci-management ~140 jobs spread across 16 repositories. Verify Merge Daily Weekly Release jobs Nightly jobs Types of Tests CI executes Unit-test, Functional, Smoke, Integration, Regression, Performance, Vulnerable tests, Code analysis tests Documentation RTD SDK API docs Publish images and binaries Docker images Fabric and fabric-ca binaries SDK artifacts npm modules mvn jar files

Current Vs Future Current State Jobs are created with freestyle project type CI scripts in a separate repo is lacking developers attention or contribution Dependency on CI team to update changes required in CI Optimize build scripts Documentation on best practices Reduce build time Improve build notifications Future State Convert fabric, fabric-ca, fabric-sdk-node, fabric-samples, fabric-sdk-java projects to pipeline Write re-use functions to optimize build scripts Improve CI documentation Cache third party software versions in packer image which dynamically loads on x86_64 build node Build notifications to Slack, RocketChat, Email Generate release notes from nightly release jobs and create a seamless release process

Freestyle vs Pipeline Freestyle Dependency on CI team to update the Job configuration CI scripts in a separate repo is lacking developers attention or contribution Complex to manage the code CI change has to test separately before push Pipeline Keep CI scripts in code repository Easy to manage the code Write Job configuration in Jenkinsfile (Groovy script) No separate testing required. More build information on pipeline stages, easy and flexible view

Deep Dive on JJB Configuration

JJB –Freestyle Job Configuration template Jobs macros Parametrize the variables

JJB – Pipeline Job Configuration JJB with Pipeline project template Jenkinsfile

Daily/Nightly Job Flow

Nightly Build Job & Release Kickoff Jobs Nightly/Daily Job  Release Jobs Nightly Build Job & Release Kickoff Jobs 1.2.1-stable – release-1.2 1.3.0-stable – master branch Release Jobs fabric-release-job fabric-binaries-x86_64 fabric-ca-release-job fabric-ca-binaries-x86_64

Q3 CI Roadmap Key FAB’s Refactor CI Jobs. (Epic#) Convert freestyle to pipeline project types. – FAB-10067 Keep CI scripts in code repo to avoid dependency from CI Integrate build notifications (email, slack/RocketChat) in CI jobs. FAB-2180 Update existing CI & Release process documentation. FAB-8461 https://ci-docs.readthedocs.io/en/latest/ Work on CI Ad-hoc requests. Continue doing build monitoring and analyze the build failures and create bugs in JIRA Explore more on Jenkins-plugins to provide more features on community builds. Key FAB’s FAB-11398 -- Docker Hub reference documentation on fabric, fabric-ca images FABB-19 -- fabric-baseimage release strategy FAB-11018 -- Update baseimage with s390x/debian:stretch in release-1.0, release-1.1 branches FAB-2180 -- Build notifications to RocketChat/Slack/Email FAB-8461 – Improve CI documentation Pipeline FAB’s (FAB-10067) FAB-11389 FAB-11392 FAB-11261

CI tasks & Bugs created by CI team Dashboard: https://jira.hyperledger.org/secure/Dashboard.jspa?selectPageId=10103

We are active on RocketChat Channels (#ci-pipeline, #fabric-ci, #infra-support) Create JIRA under FABCI Project, choose right EPIC for the task or create one Push changes to Gerrit ci-management master branch and `cr-ci` in reviewers

Fabric Build Flow

Fabric Verify Job Flow Developers Submit Patch set JJB (Freestyle) fabric-verify-build-checks-x86_64 fabric-docs-build-x86_64 fabric-smoke-tests-x86_64 fabric-verify-integration-tests-x86_64 fabric-verify-unit-tests-x86_64          Submit Patch set Verified  +1/-1 Slave – x86_64       Code Review +2 JJB (Freestyle) Build Notifications

Fabric Verify Build Process Verify Build Checks Build Images & Binaries Make basic-checks Publish Images and binaries on commit tag If succeed, post “Run DocsBuild” or “Run SmokeTest” comments DocsBuild Run tox.ini Publish RTD output to nexus log server SmokeTest Pull Images and Binaries from nexus based on the patch set commit number Run Byfn Eyfn Tests If succeed, post “Run UnitTest” and “Run IntegrationTest” comments IntegrationTest Pull Images and Binaries from Nexus based on the commit# make integration-test UnitTest make unit-tests

Fabric Merge Job Flow Maintainers Merge Patch set JJB (Freestyle) RTD fabric-merge-x86_64 Build Fabric Run “make unit-tests” fabric-merge-end-2-end-x86_64 Build Images and Binaries Run e2e_cli Run sdk-node e2e tests Run sdk-java e2e tests Run byfn eyfn tsts          Merge Patch set Slave – x86_64       Build Notifications JJB (Freestyle) RTD https://readthedocs.org/projects/hyperledger-fabric/

SDK Build Flow

SDK (Chaincode-node, sdk-node) Developers SDK (Chaincode-node, sdk-node) Verify Job Merge Job Release Job Build Images, Binaries gulp test/gulp headless/gulp test-e2e Publish unstable npm modules Publish API docs Publish stable npm modules Verified  +1/-1 Slave – x86_64       Slave – s390x Code Review +2 JJB (Freestyle) Build Notifications Maintainers After release tag is created * - Publishing npm modules happens only from x86_64