1. AI project components: Facter and Hiera
Vítor Gouveia,
IT-PES-PS

2. Agenda Recap: AI configuration components
Future life cycle development
What is Facter?
How to write custom facts
Distribute additional facts
What is Hiera?
Hiera in action
Questions?? 2

3. Recap: AI Configuration Components
Foreman – Web front-end organizes the AI configuration infrastructure
Hostgroups
Essentially the same as a cdb cluster / customization templates
Several features aggregated together
Hostgroups manifests are inside the hostgroups directory of each branch of the AI GIT repository hg_<hostgroupname>
Environments
Possibility to have:
machines with different configurations or different states
Different modules associated to each environment
Each environment maps to a git branch of the AI GIT repository
vocs_devel -> temporary branch for VOCs tests

4. Future life cycle development
Ai configuration components structure
Core Modules
Shared Modules
Individual Modules
AI Golden Branches of the AI GIT repo
Devel
Testing
Master
Custom branches should be used for particular developments and contributions only. 4

5. Future life cycle development
If you have individual modules ( not shared among other groups), you will be able to push them directly into the master branch without the intervention of the RMT 5

6. Future life cycle development
Service Managers are responsible for the hostgroup manifests and the hostgroup hieradata of their service
More about the life cycle development in future meetings 6

7. What is Facter?
Cross-platform Ruby library designed to gather information on all the nodes you will be managing with Puppet
Ex: Hardware, OS ( version information, IP Address, etc…)
executed on the puppet node and every time puppet agent runs
To see a list of facts, and those defined for use in puppet, open a shell session on a puppet node and execute facter -p
System information provided in “key => value” pairs
Exposed to puppet manifests as global variables
Ex: “$::operatingsystem”
If the defaults facts are not enough you can write your own custom facts 7

8. Facter and the AI
The information about the nodes is resolved dynamically and exposed as facts to puppet manifest
Where is the information coming from?
We have several sources of truth:
Ex: LanDB
Significant improvement over CDB
Quattor was copying information from sources into the machine’s CDB profile
We had to update CDB profile if something changed
now the information is retrieved dynamically from the source 8

9. Facter examples Landbd facts [root@vitslc6 ~]# facter -p | grep landb
landb_location =>
landb_network_domain => LCG
landb_rackname => 0000
landb_responsible_ =>
landb_responsible_name => VITOR EMANUEL GOMES GOUVEIA 9

10. How to write custom facts
Write customs facts with ruby and distribute them through the puppet master.
Ex: File redhat_release.rb
Create the folder lib/facter inside your hostgroup folder and put your rb files there.
require 'facter'
Facter.add(:redhat_release) do
setcode do
Facter::Util::Resolution.exec("cat /etc/redhat-release")
end 10

11. How to write custom facts
For the moment…
Push the changes into the vocs_devel branch and wait until they are distributed by the puppet master.
Once the custom fact are distributed
you can test them:
facter --puppet redhat_release
How to use them? $::redhat_release
More information about facts here: 11

12. Hierarchical database
Hiera Overview
Hierarchical database
Hierarchical-based lookup system
Key/value lookup tool for hostgroup-specific data
Why?
Provide parameters/input information to (shared or individual) modules
Separate data from puppet code
Hostgroup-specific data will be out of your manifests 12

13. Hiera Overview How does it work?
Puppet manifests ask for the value associated with a key
hiera(<key>,[default_value])
Hiera will search an hierarchy of sources until a value is found for the key
Hiera(<key>,[default_value]) returns a single value
But if you want to search through ALL levels of the hierarchy and return EVERY value for a specific parameter:
hiera_array(<key>,[default_value]) 13

14. Complete description of the YAML’s format:
Hiera Overview
YAML’s Format
Complete description of the YAML’s format:
---
# array
apache-packages:
- apache2
- apache2-common
- apache2-utils
# string
apache-service: apache2
# interpolated facter variable
hosts_entry: sandbox.%{fqdn}
# hash
sshd_settings:
root_allowed: "no"
password_allowed: "yes“
# alternate hash notation
sshd_settings: {root_allowed: "no", password_allowed: "yes"}
# to return "true" or "false"
sshd_settings_2: {root_allowed: no, password_allowed: yes} 14

15. A small part of the hiera.yaml hierarchy:
Hiera in action
AI Hiera Backends in the AI: YAML and GPG
Hiera’s config file is usually referred to as hiera.yaml
You can check the complete hierarchy inside a puppet node: /etc/puppet/hiera.yaml
A small part of the hiera.yaml hierarchy:
Environments -> top level location is pulled from the AI git repository.
Suppose we have a host that:
Is in the voatlas/webserver hostgroup
Is in the puppet environment vocs_devel
Contains in the apache module: $proxy_deny = hiera("apache_proxy_deny", "all")
- environments/%{environment}/hieradata/hostgroups/%{encgroup_0}/%{encgroup_1}
- environments/%{environment}/hieradata/hostgroups/%{encgroup_0}
- environments/%{environment}/hieradata/environments/%{environment}
- environments/%{environment}/hieradata/module_names/%{module_name}
- environments/%{environment}/hieradata/common 15

16. Hiera in action Regarding only the previous hiera hierarchy.
Hiera will look the following files:
hieradata/hostgroups/voatlas/webserver.yaml
hieradata/hostgroups/voatlas.yaml
hieradata/environments/vocs_devel.yaml
hieradata/module_names/apache.yaml
hieradata/common.yaml
- environments/%{environment}/hieradata/hostgroups/%{encgroup_0}/%{encgroup_1}
- environments/%{environment}/hieradata/hostgroups/%{encgroup_0}
- environments/%{environment}/hieradata/environments/%{environment}
- environments/%{environment}/hieradata/module_names/%{module_name}
- environments/%{environment}/hieradata/common 16

17. Example: configuring the high_load lemon exception
Base configuration
lxplus hostgroup: alarms only if load higher than 30, so in lxplus.yaml we put:
While in service X we are very sensitive to high load conditions and want the alarm to be triggered earlier, so we put in serviceX.yaml:
lemon_30008:
  sensor: exception
  metricname: exception.high_load
  metricclass: alarm.exception
  params:
    Correlation: "20002:1 > 2"
    MinOccurs: 10
lemon_30008:
  params:
    Correlation: "20002:1 > 30"
lemon_30008:
  params:
    Minoccurs: 2 17

18. Hiera and Facter in Action18

19. https://community.web.cern.ch/tutorial/how-store-secrets-hiera
More info about Hiera
More info about Hiera
How to store secrets in Hiera
Ai community exchange 19

20. ??????????????????????? 20