Univers Libre - ansible

Using molecule with GitLab CI

2019-04-21T00:00:00+00:00

This article follows this one where I talk about verifying and testing your Ansible roles with molecule.

As I wrote in this first article, I also configured GitLab CI to automatically run molecule tests after each push to the repository. A good documentation to start with is the official GitLab CI/CD documentation.

Here is our .gitlab-ci.yml (latest version) used on our ansible repository hosting all our roles:

---
image: quay.io/ansible/molecule:latest
services:
  - docker:dind

stages:
  - tests

before_script:
  - docker -v
  - python -V
  - ansible --version
  - molecule --version

molecule-role-common:
  stage: tests
  tags:
    - docker
  variables:
    DOCKER_HOST: "tcp://docker:2375"
    PY_COLORS: 1
  script:
    - cd roles/common/
    - molecule test
  only:
    changes:
      - roles/common/**/*

molecule-role-monitoring-tools:
  stage: tests
  tags:
    - docker
  variables:
    DOCKER_HOST: "tcp://docker:2375"
    PY_COLORS: 1
  script:
    - cd roles/monitoring-tools/
    - molecule test
  only:
    changes:
      - roles/monitoring-tools/**/*
[…]

Our repository is hosted on gitlab.com and even if we can connect our own GitLab Runners to gitlab.com instance, we decided to use the provided gitlab.com’s shared runners to get started quickly.

Those use Docker to run the jobs. So we instantiate the official molecule image (from Red Hat’s registry):

image: quay.io/ansible/molecule:latest

And since molecule also needs to spawn Docker containers, we need to start another container: docker:dind (Docker in Docker). This is done with:

services:
  - docker:dind

And by exporting the DOCKER_HOST variable to the jobs, so that molecule will be able to connect to the Docker daemon to manage its containers:

variables:
  DOCKER_HOST: "tcp://docker:2375"

Then we define one stage, called tests:

stages:
  - tests

Each jobs related to this stage will have the stage: tests key in their definition.

The before_script block is here for debugging purpose (if the molecule tests succeed locally bt fail on the runners, it may be because the tests are run with a different version of molecule).

Last, we define as many jobs as we have roles to test with molecule. Here, tags is important since it is used by GitLab CI to select the runner the job will be executed on. We need a runner with Docker, thus:

tags:
  - docker

With script keyword we specify each command the runner should run. Pretty self-explanatory:

script:
  - cd roles/monitoring-tools/
  - molecule test

And only adds kind of a condition to the job. We don’t want to run all jobs if, for instance, someone updates the README file. The job should run only if a commit introduces a change to a file in the role itself. So we restrict the execution to a change under the role’s hierarchy only with:

only:
  changes:
    - roles/monitoring-tools/**/*

For instance, this pipeline, triggered by the commit afe5929c, has spawned only one job, the molecule-role-firewall test, since only the firewall role was modified. Whereas this commit (which impacted multiple roles) triggered 7 jobs (and yes, one has failed).

If a job fails, the commiter will receive a notification by e-mail with the commit and job ID. We can check the job’s output on GitLab, to see why it has failed.

As an ending note, to test your .gitlab-ci.yml without pushing, you can install the gitlab-runner package and run it on your repository (you still need to commit your changes since `gitlab-runner will checkout the repository):

$ gitlab-runner exec docker <your job name>

Automatically generate a diagram of your infrastructure from Ansible inventory

2019-04-05T00:00:00+00:00

Another post about Ansible. This time I will talk about generating a representation of your infrastructure, for documentation purpose, using your Ansible inventory and gathered facts.

I had to update the infrastructure diagram of the Services FACiLes project, it was deprecated and, inevitably, will still be in a few months because we keep adding new containers and services. That’s a common issue in tech documentation.

Moreover, manually editing a graph isn’t a thing I usually like to do, especially when elements, relations and additional text can more easily be declared in a text file using a definition language. I found for instance this tool (not tested) which allows you to describe what you want to graph using a YAML file and then use graphviz to draw the resulting diagram. But, wait, I already have this kind of structured data, it’s in my Ansible inventory file.

I then thought about writing an equivalent tool to draw a diagram representing Ansible hosts and groups declared in the inventory and their facts and variables. Of course, someone else already had this idea, and this is called ansible-inventory-grapher. This python tool produces graphviz code that you can directly pipe to graphviz to generate a png or svg file.

As an example, here is how the graph of our infrastructure looks like:

ansible-inventory-grapher doesn’t show value of variables but you can use [this fork]((https://github.com/eking-go/ansible-inventory-grapher/tree/variables_with_value) merge request pending) to have them on your graph. You should also enable fact caching in your ansible.cfg and gather your server’s facts before running ansible-inventory-grapher. Else you will only have variables declared in host_vars/ and group_vars/.

The default jinja2 template used to generate the code suitable for graphviz prints all variables and facts, but I was only interested by some of them (those passed to the --visible-vars actually) so I slighty modified the template as follow:

--- doc/ansible-inventory-graph.default.j2
+++ doc/ansible-inventory-graph.j2
@@ -8,7 +8,7 @@
   <tr><td><b>
   <font face="Times New Roman, Bold" point-size="16">{{ node.name}}</font>
   </b></td></tr>
-{% if node.vars and showvars %}<hr/><tr><td><font face="Times New Roman, Bold" point-size="14">{% for var in node.vars|sort %}{{var}}{% if var in visible_vars %} = {{node.vars[var]}}{% endif %}<br/>{%endfor %}</font></td></tr>{% endif %}
+{% if node.vars and showvars %}<hr/><tr><td><font face="Times New Roman, Bold" point-size="14">{% for var in node.vars|sort %}{% if var in visible_vars %}{{var}} = {% if node.vars[var] is iterable and node.vars[var] is not string %}{{ node.vars[var]|map(attribute='name')|join(', ') }}{% else %}{{node.vars[var]}}{% endif %}<br/>{% endif %}{% else %}<br />{%endfor %}</font></td></tr>{% endif %}
 </table>
 >]
 {% else %}
@@ -17,7 +17,7 @@
   <tr><td><b>
   <font face="Times New Roman, Bold" point-size="16">{{ node.name}}</font>
   </b></td></tr>
-{% if node.vars and showvars %}<hr/><tr><td><font face="Times New Roman, Bold" point-size="14">{% for var in node.vars|sort %}{{var}}{% if var in visible_vars %} = {{node.vars[var]}}{% endif %}<br/>{%endfor %}</font></td></tr>{% endif %}
+{% if node.vars and showvars %}<hr/><tr><td><font face="Times New Roman, Bold" point-size="14">{% for var in node.vars|sort %}{% if var in visible_vars %}{{var}} = {% if node.vars[var] is iterable %}{{ node.vars[var]|map(attribute='name')|join(', ') }}{% else %}{{node.vars[var]}}{% endif %}<br/>{% endif %}{% else %}<br />{% endfor %}</font></td></tr>{% endif %}
 </table>
 >]
 {% endif %}{% endfor %}
@@ -26,3 +26,5 @@
   {{ edge.source|labelescape }} -> {{ edge.target|labelescape }};
 {% endfor %}
 }

Finally, I wrote a simple Makefile to easily update the graph when inventory is updated:

# Clear out default implicit rules and variables.
MAKEFLAGS += --no-builtin-rules --no-builtin-variables

.PHONY: inventory-graph

inventory-graph: doc/ansible-inventory-graph.png

doc/ansible-inventory-graph.png: inventory/hosts $(wildcard inventory/group_vars/*.yml) $(wildcard inventory/host_vars/*.yml)
	ansible -i inventory/ -m setup all >/dev/null
	~/src/ansible-inventory-grapher/bin/ansible-inventory-grapher -i inventory/ -t doc/ansible-inventory-graph.j2 -a "rankdir=LR;" \
	--visible-vars knot_zones \
	--visible-vars backup_server \
	--visible-vars munin_master \
	--visible-vars icinga2_master \
	--visible-vars icinga2_parent \
	--visible-vars backup_hosts \
	--visible-vars icinga2_monitored_sites \
	--visible-vars db_type \
	--visible-vars hosted_by \
	--visible-vars libvirt_domains \
	--visible-vars lxc_containers \
	--visible-vars ansible_architecture \
	--visible-vars ansible_default_ipv4.address \
	--visible-vars ansible_default_ipv6.address \
	--visible-vars ansible_distribution \
	--visible-vars ansible_distribution_version \
	--visible-vars ansible_memtotal_mb \
	--visible-vars ansible_processor_vcpus \
	--visible-vars ansible_virtualization_role \
	--visible-vars ansible_virtualization_type \
	all | dot -T png >$@

That way, when we add a new machine or make any change in our infra, all we need to do is make inventory-graph and commit the resulting graph. The image file is linked in our documentation in our private wiki, thus we are ensured that it is always up to date!

Ansible QA and testing with molecule

2019-03-31T00:00:00+00:00

Winter is already over in the Laurentians, and for a few weeks now ski conditions are so horribles that it’s not fun anymore and I’m sick of applying and cleaning klister on my skis. We will soon have rain and mud and flooding everywhere. April is definitely the worst month in Quebec despite sunny and longer days. But the good thing is that I suddenly have a huge amount of time left for other things. Like system administration and coding.

Thus I picked up a task from my todo list that I always wanted to do: trying molecule over our Ansible roles and playbook at FACiL. The infrastructure running behind the Services FACiLes project is setup and maintained 100% with Ansible. All our roles and playbooks are public and hosted on Gitlab.

molecule is a Python tool developed by the same team as Ansible and aims running different kind of tests over your ansible roles: linting, role execution on multiple targets and scenarios, ensure idempotence over multiple executions and running unit tests on the target.

I am not going to write a full tutorial on molecule, a lot of them already exist on Internet, though not always up to date since molecule is still a young project and in heavy development. The official documentation is a bit rough to get started and to understand the scope of molecule. A good tutorial I found for being the most up to date and accurate.

Instead, in this post I’m going to explain how I use it to test our roles, with some tips I didn’t find in the official documentation. I also played with Gitlab CI so molecule tests are run on each git push, but it will be for another post.

molecule’s files

We start by creating a molecule directory in our role:

$ molecule init scenario --verifier-name goss --driver-name docker --role-name <role>

It will create a directory named molecule/ in your role’s directory. You can have multiple scenarios for your role, stored under molecule/ directory. By default, molecule init will create the scenario default:

$ tree molecule/
molecule/
└── default
    ├── Dockerfile.j2
    ├── INSTALL.rst
    ├── molecule.yml
    ├── playbook.yml
    ├── tests
    │   └── test_default.yml
    └── verify.yml

Each file above are just here to help you to get started and can be edited as you want. Options we passed to the molecule init command are just here to alter these files. I usually prefer copying the molecule/ directory from an existing role (which is totally acceptable) since I have done some extra customization on it.

molecule.yml

molecule.yml is the only molecule configuration file. Here is the molecule.yml we use in our roles :

---
dependency:
  name: galaxy
driver:
  name: docker
lint:
  name: yamllint
platforms:
  - name: molecule-monitoring-tools
    image: geerlingguy/docker-debian9-ansible:latest
    command: ${MOLECULE_DOCKER_COMMAND:-""}
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
    privileged: true
    pre_build_image: true
provisioner:
  name: ansible
  lint:
    name: ansible-lint
  options:
    skip-tags: molecule-converge-notest
  log: true
scenario:
  name: default

molecule is designed to be very extensible. For instance, it uses the galaxy dependency resolver (dependencies are defined in the meta/main.yml file of your role) and ansible as a provisioner (this is the only one supported for now but we can imagine molecule will be able to use other configuration management tools).

We use yamllint as the linter for all YAML file of the role (including molecule’s own files!) and ansible-lint for ansible files.

For some reasons, we don’t want to run some Ansible tasks when testing the role. molecule let us pass any ansible-playbook option with the dict provisioner:options. Here we excluded tasks which have the molecule-converge-notest tag on them.

And finally we tell molecule to run the role in a Docker container. molecule supports a large range of drivers, including LXC containers, Vagrant VMs and common public cloud providers. Even if we use LXC containers on production, I choose to use the Docker driver with molecule because, first it’s more easy to setup on a personal machine, and second because this is the only supported option if we want to run molecule tests on Gitlab’s shared runners (using Docker in Docker), I will talk about it later in a second post.

Most of our roles don’t work inside a minimal Docker container, so we use geerlingguy/docker-debian9-ansible Docker image. The image provides a more complete Debian system (with systemd and common tools). It is maintained by Jeff Geerling, who wrote a lot of Ansible roles and uses it to test his own roles.

By default, molecule spawns a container named instance. It’s a bad idea since I ended up with conflicts when I tried to launch tests on multiple roles at the same time, each of them was executed in the same container. Thus I edited platforms:name key so that it bears the role’s name.

Dockerfile.j2

This file is only here if you want to build your Docker container on each test. You can use any Docker image (defined in platforms:image) and this Dockerfile will install requirements such as python, sudo, bash and ca-certificates package to run Ansible.

Since the image we use already contains those dependencies, we instruct molecule not to build another Docker image (platforms:pre_build_image). this way, we also speed up tests.

INSTALL.rst

INSTALL.rst is an informal file containing instructions to others contributors and requirements to run molecule tests. For instance, we need to have the Docker engine and docker-py installed on our machine.

playbook.yml

After molecule.yml, this is the second most important file. Once molecule has spawned the container, it will run this playbook.

Here is its content:

---
- name: Converge
  hosts: all
  roles:
    - role: <role>

Pretty simple, it applies <role> onto the spun up container. But if your role requires some variables to be set, or dependencies that aren’t declared in your role’s meta/main.yml file, you can add them here. For instance:

---
- name: Converge
  hosts: all
  roles:
    - role: <other role>
    - role: <role>
  vars:
    foo: "bar"

verify.yml and tests/test_default.yml

Finally, those two files are for unit tests. verify.yml installs Goss (the verifier tool) in the container and run tests it finds in the tests/ directory. Although it’s a very interesting subject, I didn’t dig too much into unit tests for now.

Running molecule

First, to enable bash completion:

$ eval "$(_MOLECULE_COMPLETE=source molecule)"

Running molecule without argument will show you the list of available sub-commands. I’m not going to explain all of them, and I don’t even sure what all these sub-commands do, I still learning about molecule.

Run linting tools on your role:

$ molecule lint

Test your role into a freshly created container (by executing playbook.yml on it):

$ molecule converge

molecule converge will automatically lint your code prior to run it.

At this time you can either login to the container to manually check what has been done:

$ molecule login

Or replay molecule converge if you obtained errors and have fixed them in your role.

molecule converge allows us to pass almost any ansible-playbook option. For instance, you may want to debug tasks with a specific tag:

$ molecule converge -- -vvv --tags <tag>

To destroy the running container so you can start with a new one:

$ molecule destroy

If you just want to start a container but don’t execute anything in it (for instance if you want to known the state of a file in a brand new install):

$ molecule prepare

To run idempotence check (molecule will simply run the playbook.yml a second time and check that no tasks report a changed state):

$ molecule idempotence

All these previous commands are useful while you are developing a role. Once you have done or did only minors edits (you are pretty confident that you didn’t break anything), it’s handy to run all the test suite at once:

$ molecule test

It will lint your code, start a new container, apply playbook.yml, verify idempotence, run unit tests and finally destroy the container.

[Ansible] sudo password from your password manager or agent

2019-03-29T00:00:00+00:00

Make Ansible retrieve the sudo password from my password manager. For a long time, I was wondering how to solve this issue. Ansible is my main tool to manage my own servers. All the infrastructure of the Services FACiLes project I contribute to (a future CHATONS is built with Ansible automation in mind, and even at my previous work, Ansible is used on a daily basis.

A thing that really annoyed me was that each time I run a playbook, I have to give Ansible my sudo password. Besides that, Ansible prompts for the password at an early stage of execution, even before trying to include all files or running a syntax check on them. Thus it isn’t uncommon that Ansible fails immediately after because of a mistake I made in a playbook, and I have fix it, rerun the command, reenter my password… and so on. Which is very frustrated.

I know some workarounds exist to avoid dealing with the interactive sudo password prompt: allowing Ansible to be executed without password on the servers (almost equivalent to allowing any command to be executed without password…, which is not an acceptable answer), storing your password into ansible_sudo_pass variable in a cleartext file somewhere (yes you can encrypt it with ansible-vault but then you have to give the vault password each time you run your playbook), trying to mess with except…

The simple and elegant way of doing it for me would be that Ansible gets the password from an agent, like gpg-agent, or execute an arbitrary command to get it. After a lot of research and experimentation, I found a pretty nice and clever solution on Stackoverflow (far at the bottom, it should be upvoted more than that IMHO). The idea is to combine --extra-var ansible option (to which a file descriptor can be passed, while prefixing it with @) and <() bash operator to convert a command’s stdout to a file virtual file descriptor. We end up with the following command:

$ ansible-playbook -e@<(echo "ansible_sudo_pass: $(pass Sysadmin/xxxx)") playbook.yml

This way my sudo password is retrieved from my pass’s password manager (which uses gpg-agent) and I’m not bothered anymore with Ansible’s interactive prompt \o/.

The only drawback is that I didn’t find a way to set this into the ansible.cfg or somewhere else, except writing a bash alias or a small wrapper.