1. DevOps Meetup – Infrastructure As Code

2. A distributed active/active architecture
48 locations
Canary AWS east HA/autoscaled
Canary AWS west HA/autoscaled

3. x n x m Micro-service architecture nginx tomcat/jetty
Scale independently, Maximize utilization
edge
backend
nginx
x n
saasbase_db
saasbase_analytics
esm_event_collector
hbase_regionserver
hbase_master
hbase_thrift
hadoop_datanode
hadoop_tasktracker
kafka_broker
opentsdb_tsd
opentsdb_tcollector
opsview_agent
x m
zookeepeer
tomcat/jetty 4

4. Geo Distribution

5. DevOps, OpsDev - a two way street
Developer should be aware of the underlying infrastructure
SRE engineer should be aware of the application on top
Part of the same team – same goal (uptime/SLA), different focus

6. Infrastructure as Code
First, your infrastructure is just an “external service”
Dev/QA/Stage/Prod environments parity
Use the same recipes and playbooks in all environments
Product-in-a-box : development boxes used for local development and integration testing
Deployable services
re-use deployment recipes across products
deployment code is application code
cluster level orchestration (ansible/salt)
incremental rollouts (provides zero downtime upgrades)
Observable services
monitoring and alerting are built-in into the service
Codahale metrics, JMX, NewRelic
Integrated Dashboard to view all metrics (Grafana/OpenTSDB) 6

7. It’s not all about tools. Processes matter too
Git is the place all code meets : The Code contribution guide is our “Constitution”
Commit with : code reviews from min 2 eng, 2 x thumbs up & no thumbs down
One negative vote : must discuss and agree.
One single exception allowed : Severity 1 incidents
Everyone can contribute to everything through a pull-request.
Development & debugging flows matter
Every engineer can automatically spin up a local VM with everything running on it.
Some dev are working early in the day, others late – strive to not enable them not depend on others.
If it bothers you, you’re the one motivated to fix it.
Engineer on-call is the first one to receive a PD alert
Next days after a PD alert are for fixing the issue or tuning the alert
SREs embedded in the team
Part of the engineering team (sprint planning, features kick-off when needed)
Not shared with other products/services

8. Practical example (1)

9. Practical example (2)

10. Clusters hiera blueprints YourProduct-in-a-box
Demos
Clusters hiera blueprints
YourProduct-in-a-box
CI: vagrant box used to run end-to-end deployment validation
Same blueprint as a full scale production (just the scale is different )
Monitoring
role based check registration
Opsview
APM: Grafana
integrated APM metrics 10