Automating operational procedures with Daniel Fernández Rodríguez - Akos Hencz - CERN IT-CM Group 21/04/2016 HEPiX Spring 2016 Workshop1

Outline What is Rundeck? What does Rundeck provide us? Deployment Use cases Missing things Summary 21/04/2016 HEPiX Spring 2016 Workshop2

What is Rundeck?  Automate routine operational procedures  Company -  Language Java  Apache Software License - Code on  Actively developed 21/04/2016 HEPiX Spring 2016 Workshop3

Concepts Projects: used to separate management activity. All Rundeck activities occur within the context of a project Jobs: a sequence of steps, job options, and the nodes where the steps are executed Executions: An execution is a representation of the activity of a running or completed job Plugins: extend and expand Rundeck’s functionality 21/04/2016 HEPiX Spring 2016 Workshop4

What does Rundeck provide us? Common place for implemented workflows and jobs Clean and easy-to-use Web interface Allow for scheduled jobs, cron-style Jobs reusability 21/04/2016 HEPiX Spring 2016 Workshop5

What does Rundeck provide us? Fine-grain access control  Ex: Sysadmins can only execute, Cloud Team modify + execute Traceability and auditing Delegation internal sensitive tasks to other groups without exposing credentials or procedures  Ex: Sysadmins remove node from infrastructure w/o having rights Complete log of every executed job 21/04/2016 HEPiX Spring 2016 Workshop6

All puppetized Currently working on a Puppet module to deploy Rundeck with some CERN specifics Different deployments for different services Credentials Deployment 21/04/2016 HEPiX Spring 2016 Workshop7

Deployment mod_shib Apache :443 Rundeck :4440 Tomcat :8009 rundeck-01.cern.ch 21/04/2016 HEPiX Spring 2016 Workshop8

Use cases  OpenStack Management  Project Creation  Project Quota Updates  Reporting on the Cloud  Usage and status  Health reports  Notifications  Hyper-V Patching Campaigns  VM owners  Checks and verifications  Consistency checks  GNI Tickets  Configuration  Rename physical host  Register new Puppet entities  Various scheduled synchronizations  Host Management  Enable/disable Compute Host  HW Interventions  Remove physical host  Internal  Auto backups  Reload nodes  …. 21/04/2016 HEPiX Spring 2016 Workshop9

HW Intervention on a Compute Node Disable compute node Disable nova-service Switch Alarms OFF Update associated Service-Now ticket Notifications Send to owners of affected VMs Other tasks Post new message broker Add remote AT job Save intervention details Send calendar invitation 21/04/2016 HEPiX Spring 2016 Workshop10

HW Intervention on a Compute Node Kerberos granting ticket for svcrdeck obtained successfully [INFO] Changed OS_AUTH_URL to [INFO] Changed SERVICE_ENDPOINT [INFO] Changed OS_REGION_NAME to cell12 [INFO] Appending cern.ch to the host name... [INFO] Trying to disable nova-compute on P S20073.cern.ch | Host | Binary | Status | Disabled Reason | | P S20073.cern.ch | nova-compute | disabled | [INC ] Disabled by rsantalu using Rundeck (11823) | [INFO] nova-compute sucessfully disabled on P S20073.cern.ch. TOTAL HOSTS SUCCESS ERROR [PERFORM] SUMMARY NOVA DISABLE [INFO] Using roger to disable alarms on the following hosts: P S20073 [INFO] Trying to disable roger alarms on P S [INFO] Roger alarms sucessfully disabled on P S [ { "app_alarmed": false, "appstate": "production", "expires": "", "hostname": "p s20073.cern.ch", "hw_alarmed": false, "message": "[INC ] Disabled by rsantalu using Rundeck (11823)", "nc_alarmed": false, "os_alarmed": false, … 21/04/2016 HEPiX Spring 2016 Workshop11

Register new Puppet entities Create new puppet modules & hostgroups JIRA Configuration Change ticket  Ticket status APPROVED  JIRA Web-Hook executes job via Rundeck REST API 21/04/2016 HEPiX Spring 2016 Workshop12

Register new Puppet entities Parse JIRA ticket with the given ID  Create project in Gitlab, set ACLs, user groups, LDAP links  Create puppet skeleton in project & QA branch  Register project in Foreman and other systems Close ticket 21/04/2016 HEPiX Spring 2016 Workshop13

GNI Alarm Tickets Opened to Cloud Team GNI Not assigned Mark ticket as resolved or Assign to person in ROTA 5 21/04/2016 HEPiX Spring 2016 Workshop14

Missing things… Pass data between jobs ( Share jobs between Projects ( Resume job from where error occurred ( Configure scheduled jobs to be run once (AT jobs) ( Allow user interaction ( 21/04/2016 HEPiX Spring 2016 Workshop15

Summary Rundeck is a great platform, easy to use and deploy Active development and increasing user community Adapt its behaviour to your preferences via custom plugins Valid for simple jobs but also for more complicated workflows 21/04/2016 HEPiX Spring 2016 Workshop16

Questions? Thank you for your attention! 21/04/2016 HEPiX Spring 2016 Workshop17

I like it. Where can start from? Web page: Oficial docs: Rundeck Trello Board for future ideas: Mailing list: discusshttp://groups.google.com/group/rundeck- discuss Puppet module: rundeckhttps://github.com/voxpupuli/puppet- rundeck 21/04/2016 HEPiX Spring 2016 Workshop19