1. 1 Policy Based Systems Management with Puppet Sean Dague http://dague.net

2. 2 Complexity This is the way we typically manage systems

3. 3 Challenges in Systems Management ● People are bad at repetitive process ● Institutional knowledge is often just in people's heads ● Many tools give an all / nothing approach to management

4. 4 Previous Solutions ● Image vs. Configuration – What is encoded in your base image – What is encoded in your configuration ● Configuration Solutions – Many, Many, Many vendor solutions – Cfengine – Lots of home grown solutions (even more than the backup space)

5. 5 Puppet Overview ● Approach - encodes "know good state" – Important that actions are idempotent – Turns institutional knowledge into "code" ● Mechanics – client / server system ● policy resides on server ● clients pull policy updates – Inspired by cfengine – Written in Ruby – Policy language a ruby-like DSL – Support for "facts" to customize policy based on target ● os, arch, hostname, etc

6. 6 Puppet Architecture

7. 7 Policy compiled when client requests it Client runs under existing policy if error with server Facts computed about a client on server

8. 8 Simplest Policy Example class sudo { package { sudo: ensure => latest } file { "/etc/sudoers": owner => "root", group => "root", mode => 440, require => Package["sudo"] } node default { include sudo }

9. 9 default file:/etc/sudoers package:sudo class sudo

10. 10 Common Parts of Puppet Policy ● Built in types – Package – File – Service – User – Group – Exec ● case $fact – basic conditional support based on "facts" ● Additional customization can be done via plugins

11. 11 Specific Example: IBM Grid

12. 12 OpenSim Management ● OpenSim is a 3D Open Source server implementing the 2 nd Life protocol ● Has 5 Grid services (can be on one or multiple machines) ● N Region servers (can be geographically distributed) ● IBM Grid - an IBM internal volunteer effort to run a large scale IBM Grid ● On average, bringing a new machine into the Grid was a 2 day process – Lots of issues found with subtle differences in configurations – Humans do repetition surprisingly badly

13. 13 IBM Grid Planning Session

14. 14 What's a Region Server ● A base version of Linux ● A set of dependent software ● Custom compiled version of Mono ● MySQL database, configured correctly ● Custom compiled version of OpenSim, with all Regions servers at a specific revision

15. 15 What's a Region Server ● A base version of Linux ● A set of dependent software ● Custom compiled version of Mono ● MySQL database, configured correctly ● Custom compiled version of OpenSim, with all Regions servers at a specific revision ● Add on for Monitoring ● Add on for Backups ● Add on for Crash Recovery

16. 16 What's a Region Server ● A base version of Linux ● Installation/configuration of Puppet ● A set of dependent software ● Custom compiled version of Mono ● MySQL database, configured correctly ● Custom compiled version of OpenSim, with all Regions servers at a specific revision ● Add on for Monitoring ● Add on for Backups ● Add on for Crash Recovery

17. 17 Results Reduced the 2 days of effort to ● 4 steps (takes about 15 minutes for the user) ● 1 hour of background processing in Puppet With additional wins ● Crash recovery on OpenSim process ● Upgrades of entire cluster is updating 1 file and standing back

18. 18 OpenSim Puppet Policy

19. 19 OpenSim Puppet Policy MySQL Mono Install sudo backup nagios OpenSim

20. 20 Policy Specifics ● "facts" ● Mono installation ● OpenSim installation ● OpenSim monitoring ● Server stats

21. 21 Questions? ● Puppet Website - http://reductivelabs.com/products/puppet/http://reductivelabs.com/products/puppet/ ● IT Management Podcast - http://www.redmonk.com/cote/it- management-podcast/

22. 22 Strategies for Successful Policies

23. 23 Strategies for Successful Policies ● Packages ● Files ● Services ● Users ● Groups ● Exec – Catchall – Be very careful that scripts are idempotent