1. Platform & Engineering Services CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Development Workflow of the Configuration Management Vítor Gouveia, vitor.gouveia@cern.ch IT-PES-PS

2. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Agenda Puppet 3, Git, Ibex and Grizzly New Development Workflow of the Configuration Management

3. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Puppet 3, Git,Ibex and Grizzly Puppet 3 is a reality –All the infrastructure: Puppet Masters Clients Backwards-Incompatible changes in 3.x –http://docs.puppetlabs.com/puppet/3/reference/whats_new.html#backwards- incompatible-changes-in-3xhttp://docs.puppetlabs.com/puppet/3/reference/whats_new.html#backwards- incompatible-changes-in-3x We are now using the CERN’s Central Git Service instead of the AI Git ServiceCentral Git Service Ibex – Will be soon disappear (end of the month) –If you still have machines there, please delete then and move to Grizzly Grizzly –the latest release of the Cloud Infrastructure Service https://openstack.cern.ch –User Guide https://information-technology.web.cern.ch/book/cern-private-cloud-user-guide

4. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES The new Development Workflow The old development workflow had several issues… –Major issues in pushing changes into production at different speeds –The test process was not easy –The ownership and responsibility of the changes wasn’t clear –Etc… The new development workflow mitigates several problems encountered in the first version –https://twiki.cern.ch/twiki/bin/view/AgileInfrastruct ure/ModulesManifestsDevelopmentCyclehttps://twiki.cern.ch/twiki/bin/view/AgileInfrastruct ure/ModulesManifestsDevelopmentCycle Lets see the major changes

5. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES The old punch-modules repo … … AI Git Service Central GIT Service

6. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES The old punch-modules repo master branch = master branch –the code in the master branch of each hostgroup/module was migrated into the master of each repository –A QA branch was created based on the new master branch devel branch migrated into migration_ai1437 –static branch –migration only-branch –opportunity for un-merged changes from devel could be merged into the new process

7. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Unclaimed hostgroups… All the hostgroups created due to the migration and unclaimed (nobody has claimed ownership yet) will be deleted on the 30 th of September. –A email was sent to ai-admins with a list of the hostgroups If you want to preserve your hostgroup and is unclaimed: –Open a JIRA ticket on the AI project requesting ownership egroup and the reason to keep it (personal, service, etc..)

8. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Development Workflow of the Configuration Management Roles –Service Manager (SM) –Module Maintainer (MM) –Core Module Team (CMT) –Release Manager Team (RMT) Components –Hostgroups A set of Puppet resource that describe a service, should not be reusable. –Ex: CASTOR, BI, etc… –Modules Package of reusable Puppet code. Normally used to configure software component. Ex: Apache, NTP, MySQL, etc… Individual – used in a service Shared – used in more than one service. Ex: afs, apache, base, bdii, bridged, etc… (More in the documentation)documentation

9. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Branches All the hostgroups and modules have two branches –master (production) latest release –qa (quality assurance) All your changes should pass an incubation period here –true for shared modules –recommended for individual modules You should create topic branches for new developments. You can never remove master and qa Even for individual modules and hostgroups, don’t write directly to the master branch. It is highly recommended to always create changes in QA then merge (fastforward) the commits in master.

10. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Structure of the hostgroups Git repository: it-puppet-hostgroup- code –manifests, files and templates –Before: Ex: punch-modules/hostgroups/vobox –After: Ex: it-puppet-hostgroup-vobox/code –Base class should always be prefixed with the hg_ (ex : hg_vobox) data/hostgroup –Hieradata for the top-level hostgroup and all the descendants –Before: Ex: punch-modules/hieradata/hostgroups/vobox.yaml punch-modules/hieradata/hostgroups/bi/inter.yaml –After: Ex: it-puppet-hostgroup-vobox/data/hostgroup/vobox.yaml it-puppet-hostgroup-bi/data/hostgroup/bi/inter.yaml data/fqdns –Host level hieradata –Before: Ex: punch-modules/hieradata/fqdns/lxplus0233.cern.ch.yaml –After: Ex: it-puppet-hostgroup-bi/data/fqdns/lxplus0233.cern.ch.yaml

11. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Creation of a top-level hostgroup 1.Open a ticket Configuration Release Management JIRA projectConfiguration Release Management JIRA project ( create an issue ) 2.Create an empty git repository in the CERN’s Central Git ServiceCentral Git Service –it-puppet-hostgroup- ACL –ai-config-team and SM/MM can modify master and qa ai-config-team = admins of Configuration Management Service –New branches can be created by: ai-config-team ai-admins (all the users of the Configuration Management service) –propose patches to the MM via topic branches MM/SM –“official” developments through topic branches 3.Go to Foreman and create your top-level hostgroup Issue Type: Add Configuration Element Summary: name of the hostgroup Addition Type: hostgroup Why needed: Top-level hostgroup for service X Responsible FE: Which FE is going to be the MM Labels: Eases the searching and filtering

12. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Creation of a top-level hostgroup Foreman: –Become an hostgroup manager: The process to become an hostgroup manager is described in CERN Configuration Management System User GuideCERN Configuration Management System User Guide –Now: If the hostgroup exists already you need to ask to the Configuration team to be hostgroup manager of the hostgroup X You can create a new hostgroup but you can’t give permissions to other users –Future 2/3 weeks: Integration between egroups and Foreman. This means that the access/permissions will be given to egroups instead of users

13. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Structure of the modules Git repository: it-puppet-module- code –manifests, files and templates –Before: Ex: punch-modules/modules/afs –After: Ex: it-puppet-module-afs/code/ data –Hieradata for the module –Before: punch-modules/hieradata/module_names/lemon.yaml –After: it-puppet-module-lemon/data/lemon.yaml

14. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Creation of a new module –Issue type: Add Configuration Element –Summary: name –Addition Type: Individual/Share –Origin: CERN/third-party –Why needed: Description why you need the module –Responsible FE: Which FE is going to be the MM –Labels: Eases the searching and filtering Open a ticket Configuration Release Management JIRA project (create an issue)Configuration Release Management JIRA project

15. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Changes in invidual modules/hostgroups create a topic branch. Why? –production stays clean –test a new code –squash and tweak the history when merging the new feature into production merge the topic branch into qa (testing time) merge the qa into master Delete unnecessary environments and merged branches –first the environment and then topic branch

16. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Changes in shared modules Same process and individual changes but the changes need to stay in qa for a while before being merged into production –Open a ticket Configuration Release Management JIRA project (create an issue)Configuration Release Management JIRA project Issue Type: Configuration Change Summary: Description of the change merged in qa Change Type: New feature Bug Origin: Name of the branch has been merged Assign: Automatic Proposed date for production: date when you plan to merge the change into master (default 1 week) Changelog: Useful information for the SMs Labels: eases the search later on Upgrade list: An optional

17. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Changes in shared modules Ticket states –Initial State: Open –Broken: If a problem is found in QA by someone (SM, MM or even users) –Closed: The changed has been merged into production On the proposed date (to be put into production) the changes should be pushed into Master branch by the MM

18. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Other aspects of the modules As much as possible: –only one module for different use cases People working and using shared modules should work as team to improve and maintain the shared module MM are the responsible to write the golden environments (later about the environments Every one should contribute to make the module reusable and flexible as possible MM should be always happy to receive improvements request! If you are an user of a given module and you think that something is missing or wrong –JIRA ticket: Agile Infrastructure JIRA project with a detailed explanation of your requestAgile Infrastructure JIRA project –MM will analyse your request and provide the necessary feedback

19. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Hieradata Hieradata of the environments, hardware and operating systems is located in it- puppet-common –hieradata and maintained by the ai-config-team it-puppet-common-hieradata

20. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Environments Environment ≠ new branch collection of modules and hostgroups definitions at different level (branch) –Basically you define: default branch to be used (normally master or qa) and the environment will use Overrides –which branch should be used for specific modules, hostgroups or even common components (site.pp or hieradata) Notifications –Who to inform in case the problems and what are the overrides Git repository: it-puppet-environments

21. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Golden enviroments QA (Quality Assurance) –mandatory for shared changes –topics branches are merged with QA –machines should be here to test the impact of new changes Production –Stable and production environment –machines in this environment are going to always have successful puppet runs –controlled environment adding new environment –ai-create-environment-metadata (aiadm)

22. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Environments Yaml file –name of the Puppet environment will match the name of the file (ex: training.yaml) –only alphanumeric characters and underscore –Descriptive (ex: AI-123) --- default: master notifications: ai-config-team@cern.ch overrides: modules: foreman: ai123 puppet: ai123 hostgroups: punch: ai123test

23. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Workflow for the development of new features for Apache (example) 1.Clone the git repository of the Apache module – git clone https://:@git.cern.ch/kerberos/it-puppet-module-apachehttps://:@git.cern.ch/kerberos/it-puppet-module-apache 2.Create a new branch in the apache repository for the new feature –git checkout -b newfeature qa –do your changes and commit –git push origin newfeature 3.Create a new environment for our hostgroup based on qa that uses the branch newfeature –git clone https://:@git.cern.ch/kerberos/it-puppet-environments –cd it-puppet-environments –ai-create-environment-metadata -e testNewfeature -n ai-config- team@cern.ch -m afs:newfeature –git add testnewfeature.yaml –git commit -m “Add environment testNewfeature” –git pull –rebase origin master –git push 4.Move some machines to the environment to test your changes

24. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Workflow for the development of new features for Apache (example) 5.When you are happy merge with QA: –git checkout qa –git pull –rebase origin qa –git checkout newfeature –git rebase qa –git checkout qa –git rebase newfeature –git rebase -i origin/qa (opportunity to squash your commit) –git push origin qa 6.Open a ticket (previous described) 7.Merge your changes into master when accepted and the time is right –git checkout master –git pull –rebase origin master –If there are other changes in qa push only your changes: git merge –no-ff –If your changes are the latest ones: git rebase qa –git push origin master 8.Clean up everything (delete the temporary environment and after the temporary branch)

25. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Yaml file (example) --- default: qa notifications: ai-config-team@cern.ch overrides: modules: afs: newfeature

26. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES Development Workflow of the Configuration Management All the details can be found: https://twiki.cern.ch/twiki/bin/view/AgileInfrastructure /ModulesManifestsDevelopmentCycle (temporary) https://twiki.cern.ch/twiki/bin/view/AgileInfrastructure /ModulesManifestsDevelopmentCycle The official documentation will contain all the details of the development cycle https://information-technology.web.cern.ch/book/cern- configuration-management-system-user-guide Any problem with documentation -> snow ticket Several examples about … CERN Puppet development guidelines delete hostgroups, modules and environments, Clone hostgroups and modules FAQs etc…

27. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES One last thing… The QA environment is a very important part of the development workflow –You should/must have machines in QA: to test new changes coming from the shared modules –you can more easily identify possible problems If you identify any kind of bugs/problems with the shared modules –Open a ticket!!! It is very important part of process that everyone is aware of what is going to be in production and you stop the process before your production machines are broken –It will be easier in the future to know what is coming to production. (more about this in future meetings)

28. CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t PES ???????????????????????