Converting Hiera 3 to 5 with hi5er

Converting Hiera 3 to 5 with hi5er

Written: 2017-07-24
Author: WhatsARanjit
Links:

The setup

Hiera 3 uses a yaml configuration file that spells out your backends, configs for each backend, and finally your hierarchy. This hiera.yaml file was a global configuration which applied to all environments for a Puppet architecture.

Hiera introduces the ability to have a global, environment, and module level hierarchy. Furthermore, within each level, we can confgure more about each layer. For example, on the environment level, we are able to do something like this:

  1. Check the yaml backend with these paths on disk
  2. Check the eyaml backend with these paths on disk
  3. Check the yaml backend again with these other paths on disk.

Previously, backends were done in series. That is, everything eyaml would happen before/after everything yaml. Also, there was only one set of paths on disk to check for each backend. So in short, more flexibility; more configuration. Should you choose to use Hiera 5, you need to change your hiera.yaml into a new format that allows you to pass more configurations. The article above shows you examples of what the end product of a rewrite might look like, but it seems like something that could be done programmatically. After all, we’re rewriting a YAML file to another YAML file.

Hi5er

So that’s pretty much what hi5er does. The conversation does everything explicitly, meaning it doesn’t use the defaults ability in Hiera 5. This is so you can see all configuration at all parts. It becomes easier to change configurations when they are all there, instead marked by a default elsewhere.

Converting YAML to YAML seemed easy enough, but ruby’s YAML libraries do this wierd thing where you can YAML.load a value of true and it becomes a Boolean in ruby. But when you .to_yaml that data hash, it prints it as 'true', a String. Also, YAML prints with double-quotes sometimes when you need single-quotes, and vice-versa. So mostly the script battles that. So let’s start with a simple YAML:

---
:backends:
  - yaml
:hierarchy:
  - "nodes/%{::trusted.certname}"
  - common

:yaml:
  :datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"

This is an easy, typical hiera.yaml file. Conversion uses a rake task and looks like this:

[root@master ~/hi5er]# rake
---
version: 5
hierarchy:
- name: Yaml backend
  data_hash: yaml_data
  paths:
  - "nodes/%{::trusted.certname}.yaml"
  - 'common.yaml'
  datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"

Let’s look at a more complex config file with 2 backends:

---
:backends:
  - yaml
  - eyaml
:hierarchy:
  - "nodes/%{::trusted.certname}"
  - common

:yaml:
  :datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"

:eyaml:
  :datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"
  :pkcs7_private_key: /path/to/private_key.pkcs7.pem
  :pkcs7_public_key:  /path/to/public_key.pkcs7.pem
  :extension: 'yaml'

…comes out as …

[root@master ~/hi5er]# rake
---
version: 5
hierarchy:
- name: Yaml backend
  data_hash: yaml_data
  paths:
  - "nodes/%{::trusted.certname}.yaml"
  - 'common.yaml'
  datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"
- name: Eyaml backend
  lookup_key: eyaml_lookup_key
  paths:
  - "nodes/%{::trusted.certname}.yaml"
  - 'common.yaml'
  datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"
  options:
    pkcs7_private_key: "/path/to/private_key.pkcs7.pem"
    pkcs7_public_key: "/path/to/public_key.pkcs7.pem"
    extension: yaml

Extra configurations for each backend will appear under the options key in Hiera 5. You’ll notice that the conversion prints out data_hash for the YAML backend and lookup_key for EYAML. There are 3 types of Hiera backends, which you can follow up on here. The script recognizes the following:

Anything that outside of this gets printed as a hiera3_backend. The reason being, I didn’t feel like cataloging all the backend that are out there. I’m not sure which ones are used the most and which are edge cases. The backends above are the ones that are now built into Hiera. If you need to let hi5er know that your custom backend is a lookup_key or a data_dig, you can set some configuration options in the Rakefile:

conversion = HieraFiver.new(config_file)
conversion.backends_data_hash  = (['yaml', 'json', 'hocon', 'someotherhash'])
conversion.backends_lookup_key = (['eyaml', 'someotherlookup'])
conversion.backends_data_dig   = (['somedatadig'])

Check out the full rake task with rake -T. The script looks up hiera_config in Puppet and reads that file by default. Using the long-form of the command you can specify an alternate file location for testing.

[root@master ~]# rake convert[/path/to/other.yaml]