1. Glenn Buckholz glenn.buckholz@coveros.com
Title
Month Year
DevOps, CI, Opensource and the Cloud
Brown Bag Presentation – 29 March 2013
Glenn Buckholz

2. Agenda What did I do? What’s it good for? Demo Architecture Workflow
Title
Month Year
Agenda
What did I do?
What’s it good for?
Demo
Architecture
Workflow
Orchestration
Cost vs Velocity
Data Analysis
What is Coveros doing with it for customers?

3. Title
Month Year
What did I do?
Combined several technologies that Coveros is familiar with plus Puppet to demonstrate a full implementation of CI (Continuous Integration)
This could be titled “Everything I wanted to do on Forge.mil but am not allowed.”
Specifically, I cobbled together:
Jenkins
Puppet
Puppet master
EC2
Linux
CURL

4. Title
Month Year
Demo

5. Major Points How is this DevOps?
Title
Month Year
Major Points
How is this DevOps?
Releases are engineered, the systems engineer is simply pushing the button.
If the list of manual installation instructions is more than one paragraph there is more engineering work to do. Code is the documentation.
Operations people now govern the access, developers govern the how.
Developers now have the job of convincing operations to run their deployment code.
Test results
Scans
Watching an actual deploy

6. Title
Month Year
Architecture

7. Workflow Developer commits code. Jenkins detects code change.
Title
Month Year
Workflow
Developer commits code.
Jenkins detects code change.
Proper RPMs are built and distributed to repositories.
Jenkins contacts EC2 and spins up blank instances.
Puppet is delivered to the blank instances.
Jenkins works with the Puppet Master for orchestration.
Jenkins confirms that the application is running.
Jenkins fires off automated tests.
Upon success, everything is torn down.
Failure leaves the instance dormant for a period of time for inspection.

8. Code Commit – Different Approaches
Title
Month Year
Code Commit – Different Approaches
Something has to trigger the testing. A code commit is typically a significant enough change to require some measure of retesting.
By convention, code commits should be complete and significant.
Each commit now carries a cost.
This should be enforced by training.
For larger projects with large volumes of commits, a timed interval is more appropriate.
Like Tinderbox for Mozilla, only virtual.

9. Title
Month Year
RPMs … Packaging
To move quickly code must be delivered in a consistent format.
Copying files is too unpredictable and error-prone.
Permissions
Location
Clean up of old file
Dependency checking
Any of the following are good choices
RPM – Centos, RHEL
Emerge/Portage - Gentoo
Deb – Debian, Ubuntu
MSI - Windows

10. Cloud Provider The main issue is elasticity.
Title
Month Year
Cloud Provider
The main issue is elasticity.
Number of servers will vary.
Tests will be running in parallel.
Need a way to catalog all the machines
Assign metadata to VMs
Ability to group VMs and test results by deployment
Automatically mapping code commits to cloud deployments is necessary for proper bookkeeping.

11. OS templates Each OS must originate with a standard template.
Title
Month Year
OS templates
Each OS must originate with a standard template.
All change must be managed by Puppet or some other deployment language
If the OS is not standard too much time will be spent engineering around OS peculiarities
CM must be mastered
Every change to a standard OS must be recorded.
Every change must be programmed, no hand-jamming
All deployments are now hands-off
You can almost get Continuous Delivery for free.
OS Customizations for only two reasons
Integrate into the Jenkins/Puppet
Integrate into the cloud provider from a standard distribution

12. Automated Testing Automated testing must be an established practice.
Title
Month Year
Automated Testing
Automated testing must be an established practice.
You cannot perform the volume of tests required for CI efficiently without some sort of automated check to separate the good changes from the bad changes.
The cost and practice must be understood within the development and testing organization.
Testing resources must be able to handle max capacity.
Tests must be able to run in parallel against different targets.
Scalability must be designed into the test framework and the individual tests.

13. Disposition of the Systems
Title
Month Year
Disposition of the Systems
Test reporting and the cloud provider must be linked through a dashboard.
Manual testers must be able to verify failures.
Testers must have access to the broken running system
Access will allow reproducibility and verification
Developers must be able to identify broken VMs
Access to the app to see the unexpected behavior
Access to the underlying OS to view logs and troubleshoot

14. Title
Month Year
Orchestration
This is the combination of architectural components and services in a meaningful fashion to produce a running system.
For this example:
Database
App server
Database initialization
App server initialization
Coordinated starting of all the services

15. Costs vs Velocity Velocity comes with parallelism.
Title
Month Year
Costs vs Velocity
Velocity comes with parallelism.
Tests running in parallel
Code being tested in a pipelined fashion.
Cost
Number of testing machines and infrastructure.
Number of parallel instances x machines per instances
Storage – keeping failed instances around
Budget and Team size
Even with infinite money if you have a team of 5, the 400th VM is likely only giving very little marginal utility.
When money is the constraint, developer and tester habits must best utilize the limited VMs wisely.

16. Title
Month Year
Data Analysis
In order to move quickly the right people must get the right data.
Test Results
Quick access to the failed test
Quick access a screen shot of the failure
Ability to assign errors to the proper developers
Integrated bug tracking.
Server Analysis
Developers need to zero in on failed servers quickly
Developers need to link failures to their code quickly
Developers need to separate system/OS failures from functional errors.

17. What is Coveros Doing with it for Customers?
Title
Month Year
What is Coveros Doing with it for Customers?
Training and Process improvement
Being the exemplar on a project to move this practice to other projects.
Showing the change in the cost to test a feature within an organization
Implementing pieces of the CI to show marginal improvements.
Using the gains marginal gains to sell the whole process.

18. Title
Month Year
Level of Effort
The demonstration needs to be customized for each environment.
Very few projects that I have dealt with are “the same”. While this demo can be a guide, I don't believe it can be generalized to a solution, each environment must be customized.
This implementation required research into the APIs and inner workings of several complicated software suites and services.
The engineering effort is commensurate with the complexity of the architecture.
Now your deploys are engineer, this truly brings the developer into the Ops part of DevOps.

19. Why Link All These Technologies Together?
Title
Month Year
Why Link All These Technologies Together?
The industry is moving this way in various forms.
AWS templates
Service Mesh
JuJu
Hadoop
Flexiant
Even though the products are varied, they all share general themes.
Inventory
Relational Mapping
OS Templates
Virtual Network Capabilities
If you know one you can have a feel for them all.

20. Title
Month Year
Thoughts? Questions?
Thanks for your time.