Getting to Push Button Deploys Moovweb January 19, 2012

Things Everyone Wants

Reality … is it done yet??? … is it done yet??? … is it done yet???

Gotta Have It! PROS Increase velocity of change/responsiveness Get Operations out of the loop Buzzword compliance – aka Shiny!!!

PROS Increase velocity of change/responsiveness

PROS – Ops Gets Out of Loop Less work for Operations Developers feel more in control

Gotta Have It! CONS Increase velocity of change Shared responsibility Not always easy

Make It So...

Components

Components - MoovStack

CI Jenkins ( maiden name is Hudson ) Checks out a deploy branch from git Runs a few sanity checks Packages software Uploads to software repo Tags build in git

Jenkins nkins- ci.org/

Jenkins

Rake

Packaging But it's sooooo hard save yourself first.....

Using FPM

Building a Gem

Right Tool Packages Something that does not change often Don't need to roll back quickly Don't need more than 1 version at a time You cache external repos or limit to just yours

Right Tool Gems Fast roll back – gem cache Easy setup of gem server Need more than 1 version at a time Bundler for dependencies Bundle exec for peace of mind

Configuration Management Puppet everything from bare metal/base image Seperate data from manifests with an ENC Max troubleshooting time is 20 minutes Amazon free ”chaos monkey” service

Puppet – Start with a sane base

ENC – Use Groups! om/guides/installing_das hboard.html

ENC – Node Entries Inherit

ENC – Scripting Additions

ENC - Output

Manifests Contain ENC Variables

Templates Contain ENC Variables

Puppet - Tips Execs hurt – try to avoid them Recursive directories – use vcsrepo instead Write your own types – just restart in between Run mongrel or passenger Use behind a load balancer for fun Use reporting, but take the DB off of your master Write your own facts and save lots of time

Puppet - Scaling

Mcollective – Do It NOW!! Orchestration framework Crazy Parallel Write more ruby code... Clients and Agents

Mcollective – Crazy Parallel 6380 nrpe checks across 370 hosts in 3.8 seconds c/devops/tree/master/mco llective/nrpe

Use Facts and Classes to Filter

Mcollective – Custom Agent

Mcollective – Custom Client

Mcollective - Tips Remeber you have a machine gun pointed at your foot Wrap command in IO.popen and log everything Log to syslog Use ApacheMQ For EC2 tune your keep alives (registerinterval) Remember you have a machine gun pointed at your foot

Mcollective - Scaling

Sinatra Web app in 5 minutes Much more minimal than rails Written in Ruby so I can interface with Mcollective easily com/

Sinatra – Addons sinatra-respond_to - useful as API Pony – send from Sinatra Twitter Bootstrap – make things pretty

Public Example guec/mco_dashboard _example

Public Example

Really Simple

Sinatra - Tips Run a few behind Nginx Use helpers whenever possible Log every push Use it to integrate all your data

Actual Push Procedure

Dashboard Functionality Push software Audit software Real-time monitoring check Performance data integration Log data integration Troubleshooting tools for developers Utilities for non-technical staff Open trouble tickets Cost information (in progress) Userful links to other data sources

Oppy – an example

Thanks Michael Catlin (organizer) Tarun Desikan (giving me free reign/enough rope) R.I. Pienaar ( Mcollective author and helpful guy) Andrew Farmer ( for helping to get this going )

Links IRC #puppet,#mcollective,#devops

Author Chris Mague IRC: bosszaru usually on #gaijin