AI project components: Facter and Hiera Vítor Gouveia, vitor.gouveia@cern.ch IT-PES-PS

Agenda Recap: AI configuration components Future life cycle development What is Facter? How to write custom facts Distribute additional facts What is Hiera? Hiera in action Questions?? 2

Recap: AI Configuration Components Foreman – Web front-end organizes the AI configuration infrastructure Hostgroups Essentially the same as a cdb cluster / customization templates Several features aggregated together Hostgroups manifests are inside the hostgroups directory of each branch of the AI GIT repository hg_<hostgroupname> Environments Possibility to have: machines with different configurations or different states Different modules associated to each environment Each environment maps to a git branch of the AI GIT repository vocs_devel -> temporary branch for VOCs tests

Future life cycle development Ai configuration components structure Core Modules Shared Modules Individual Modules AI Golden Branches of the AI GIT repo Devel Testing Master Custom branches should be used for particular developments and contributions only. 4

Future life cycle development If you have individual modules ( not shared among other groups), you will be able to push them directly into the master branch without the intervention of the RMT 5

Future life cycle development Service Managers are responsible for the hostgroup manifests and the hostgroup hieradata of their service More about the life cycle development in future meetings http://cern.ch/go/8ZGQ 6

What is Facter? Cross-platform Ruby library designed to gather information on all the nodes you will be managing with Puppet Ex: Hardware, OS ( version information, IP Address, etc…) executed on the puppet node and every time puppet agent runs To see a list of facts, and those defined for use in puppet, open a shell session on a puppet node and execute facter -p System information provided in “key => value” pairs Exposed to puppet manifests as global variables Ex: “$::operatingsystem” If the defaults facts are not enough you can write your own custom facts 7

Facter and the AI The information about the nodes is resolved dynamically and exposed as facts to puppet manifest Where is the information coming from? We have several sources of truth: Ex: LanDB https://twiki.cern.ch/twiki/bin/view/AgileInfrastructure/SourcesOfTruth Significant improvement over CDB Quattor was copying information from sources into the machine’s CDB profile We had to update CDB profile if something changed now the information is retrieved dynamically from the source 8

Facter examples Landbd facts [root@vitslc6 ~]# facter -p | grep landb landb_location => 0000 0-0000 landb_network_domain => LCG landb_rackname => 0000 landb_responsible_email => vitor.gouveia@cern.ch landb_responsible_name => VITOR EMANUEL GOMES GOUVEIA 9

How to write custom facts Write customs facts with ruby and distribute them through the puppet master. Ex: File redhat_release.rb Create the folder lib/facter inside your hostgroup folder and put your rb files there. require 'facter' Facter.add(:redhat_release) do setcode do Facter::Util::Resolution.exec("cat /etc/redhat-release") end 10

How to write custom facts For the moment… Push the changes into the vocs_devel branch and wait until they are distributed by the puppet master. Once the custom fact are distributed you can test them: facter --puppet redhat_release How to use them? $::redhat_release More information about facts here: https://puppetlabs.com/blog/facter-part-1-facter-101/ 11

Hierarchical database Hiera Overview Hierarchical database Hierarchical-based lookup system Key/value lookup tool for hostgroup-specific data Why? Provide parameters/input information to (shared or individual) modules Separate data from puppet code Hostgroup-specific data will be out of your manifests 12

Hiera Overview How does it work? Puppet manifests ask for the value associated with a key hiera(<key>,[default_value]) Hiera will search an hierarchy of sources until a value is found for the key Hiera(<key>,[default_value]) returns a single value But if you want to search through ALL levels of the hierarchy and return EVERY value for a specific parameter: hiera_array(<key>,[default_value]) 13

Complete description of the YAML’s format: Hiera Overview YAML’s Format Complete description of the YAML’s format: http://www.yaml.org/ --- # array apache-packages: - apache2 - apache2-common - apache2-utils # string apache-service: apache2 # interpolated facter variable hosts_entry: sandbox.%{fqdn} # hash sshd_settings: root_allowed: "no" password_allowed: "yes“ # alternate hash notation sshd_settings: {root_allowed: "no", password_allowed: "yes"} # to return "true" or "false" sshd_settings_2: {root_allowed: no, password_allowed: yes} 14

A small part of the hiera.yaml hierarchy: Hiera in action AI Hiera Backends in the AI: YAML and GPG Hiera’s config file is usually referred to as hiera.yaml You can check the complete hierarchy inside a puppet node: /etc/puppet/hiera.yaml A small part of the hiera.yaml hierarchy: Environments -> top level location is pulled from the AI git repository. Suppose we have a host that: Is in the voatlas/webserver hostgroup Is in the puppet environment vocs_devel Contains in the apache module: $proxy_deny = hiera("apache_proxy_deny", "all") - environments/%{environment}/hieradata/hostgroups/%{encgroup_0}/%{encgroup_1} - environments/%{environment}/hieradata/hostgroups/%{encgroup_0} - environments/%{environment}/hieradata/environments/%{environment} - environments/%{environment}/hieradata/module_names/%{module_name} - environments/%{environment}/hieradata/common 15

Hiera in action Regarding only the previous hiera hierarchy. Hiera will look the following files: hieradata/hostgroups/voatlas/webserver.yaml hieradata/hostgroups/voatlas.yaml hieradata/environments/vocs_devel.yaml hieradata/module_names/apache.yaml hieradata/common.yaml - environments/%{environment}/hieradata/hostgroups/%{encgroup_0}/%{encgroup_1} - environments/%{environment}/hieradata/hostgroups/%{encgroup_0} - environments/%{environment}/hieradata/environments/%{environment} - environments/%{environment}/hieradata/module_names/%{module_name} - environments/%{environment}/hieradata/common 16

Example: configuring the high_load lemon exception Base configuration lxplus hostgroup: alarms only if load higher than 30, so in lxplus.yaml we put: While in service X we are very sensitive to high load conditions and want the alarm to be triggered earlier, so we put in serviceX.yaml: lemon_30008:   sensor: exception   metricname: exception.high_load   metricclass: alarm.exception   params:     Correlation: "20002:1 > 2"     MinOccurs: 10 lemon_30008:   params:     Correlation: "20002:1 > 30" lemon_30008:   params:     Minoccurs: 2 17

Hiera and Facter in Action 18

https://community.web.cern.ch/tutorial/how-store-secrets-hiera More info about Hiera More info about Hiera http://docs.puppetlabs.com/hiera/1/index.html How to store secrets in Hiera https://community.web.cern.ch/tutorial/how-store-secrets-hiera Ai community exchange 19

??????????????????????? 20