1. Overview on CI Use JJB (Jenkins Job Builder) to manage Jenkins jobs.
Supports x86_64, s390x build platforms.
Packer image to cache the build artifacts and load them in x86_64 build nodes.
~140 jobs spread across 16 repositories.
Verify
Merge
Daily
Weekly
Release jobs
Nightly jobs
Types of Tests CI executes
Unit-test, Functional, Smoke, Integration, Regression, Performance, Vulnerable tests, Code analysis tests
Documentation
RTD
SDK API docs
Publish images and binaries
Docker images
Fabric and fabric-ca binaries
SDK artifacts
npm modules
mvn jar files

2. Current Vs Future Current State
Jobs are created with freestyle project type
CI scripts in a separate repo is lacking developers attention or contribution
Dependency on CI team to update changes required in CI
Optimize build scripts
Documentation on best practices
Reduce build time
Improve build notifications
Future State
Convert fabric, fabric-ca, fabric-sdk-node, fabric-samples, fabric-sdk-java projects to pipeline
Write re-use functions to optimize build scripts
Improve CI documentation
Cache third party software versions in packer image which dynamically loads on x86_64 build node
Build notifications to Slack, RocketChat,
Generate release notes from nightly release jobs and create a seamless release process

3. Freestyle vs Pipeline Freestyle
Dependency on CI team to update the Job configuration
CI scripts in a separate repo is lacking developers attention or contribution
Complex to manage the code
CI change has to test separately before push
Pipeline
Keep CI scripts in code repository
Easy to manage the code
Write Job configuration in Jenkinsfile (Groovy script)
No separate testing required.
More build information on pipeline stages, easy and flexible view

4. Deep Dive on JJB Configuration

5. JJB –Freestyle Job Configuration
template
Jobs
macros
Parametrize the variables

6. JJB – Pipeline Job Configuration
JJB with Pipeline project
template
Jenkinsfile

7. Daily/Nightly Job Flow

8. Nightly Build Job & Release Kickoff Jobs
Nightly/Daily Job  Release Jobs
Nightly Build Job & Release Kickoff Jobs
1.2.1-stable – release-1.2
1.3.0-stable – master branch
Release Jobs
fabric-release-job
fabric-binaries-x86_64
fabric-ca-release-job
fabric-ca-binaries-x86_64

9. Q3 CI Roadmap Key FAB’s Refactor CI Jobs. (Epic#)
Convert freestyle to pipeline project types. – FAB-10067
Keep CI scripts in code repo to avoid dependency from CI
Integrate build notifications ( , slack/RocketChat) in CI jobs. FAB-2180
Update existing CI & Release process documentation. FAB-8461
Work on CI Ad-hoc requests.
Continue doing build monitoring and analyze the build failures and create bugs in JIRA
Explore more on Jenkins-plugins to provide more features on community builds.
Key FAB’s
FAB Docker Hub reference documentation on fabric, fabric-ca images
FABB fabric-baseimage release strategy
FAB Update baseimage with s390x/debian:stretch in release-1.0, release-1.1 branches
FAB Build notifications to RocketChat/Slack/
FAB-8461 – Improve CI documentation
Pipeline FAB’s (FAB-10067)
FAB-11389
FAB-11392
FAB-11261

10. CI tasks & Bugs created by CI team
Dashboard:

11. We are active on
RocketChat Channels (#ci-pipeline, #fabric-ci, #infra-support)
Create JIRA under FABCI Project, choose right EPIC for the task or create one
Push changes to Gerrit ci-management master branch and `cr-ci` in reviewers

12. Fabric Build Flow

13. Fabric Verify Job Flow Developers Submit Patch set JJB (Freestyle)
fabric-verify-build-checks-x86_64
fabric-docs-build-x86_64
fabric-smoke-tests-x86_64
fabric-verify-integration-tests-x86_64
fabric-verify-unit-tests-x86_64
         Submit Patch set
Verified  +1/-1
Slave – x86_64
   
Code Review +2
JJB
(Freestyle)
Build Notifications

14. Fabric Verify Build Process
Verify Build Checks
Build Images & Binaries
Make basic-checks
Publish Images and binaries on commit tag
If succeed, post “Run DocsBuild” or “Run SmokeTest” comments
DocsBuild
Run tox.ini
Publish RTD output to nexus log server
SmokeTest
Pull Images and Binaries from nexus based on the patch set commit number
Run Byfn Eyfn Tests
If succeed, post “Run UnitTest” and “Run IntegrationTest” comments
IntegrationTest
Pull Images and Binaries from Nexus based on the commit#
make integration-test
UnitTest
make unit-tests

15. Fabric Merge Job Flow Maintainers Merge Patch set JJB (Freestyle) RTD
fabric-merge-x86_64
Build Fabric
Run “make unit-tests”
fabric-merge-end-2-end-x86_64
Build Images and Binaries
Run e2e_cli
Run sdk-node e2e tests
Run sdk-java e2e tests
Run byfn eyfn tsts
         Merge Patch set
Slave – x86_64
   
Build Notifications
JJB
(Freestyle)
RTD

16. SDK Build Flow

17. SDK (Chaincode-node, sdk-node)
Developers
SDK (Chaincode-node, sdk-node)
Verify Job
Merge Job
Release Job
Build Images, Binaries
gulp test/gulp headless/gulp test-e2e
Publish unstable npm modules
Publish API docs
Publish stable npm modules
Verified  +1/-1
Slave – x86_64
   
Slave – s390x
Code Review +2
JJB
(Freestyle)
Build Notifications
Maintainers
After release tag is created
* - Publishing npm modules happens only from x86_64