Ansible QA and testing with molecule

Winter is already over in the Laurentians, and for a few weeks now ski conditions are so horribles that it's not fun anymore and I'm sick of applying and cleaning klister on my skis. We will soon have rain and mud and flooding everywhere. April is definitely the worst month in Quebec despite sunny and longer days. But the good thing is that I suddenly have a huge amount of time left for other things. Like system administration and coding.

Thus I picked up a task from my todo list that I always wanted to do: trying molecule over our Ansible roles and playbook at FACiL. The infrastructure running behind the Services FACiLes project is setup and maintained 100% with Ansible. All our roles and playbooks are public and hosted on Gitlab.

molecule is a Python tool developed by the same team as Ansible and aims running different kind of tests over your ansible roles: linting, role execution on multiple targets and scenarios, ensure idempotence over multiple executions and running unit tests on the target.

I am not going to write a full tutorial on molecule, a lot of them already exist on Internet, though not always up to date since molecule is still a young project and in heavy development. The official documentation is a bit rough to get started and to understand the scope of molecule. A good tutorial I found for being the most up to date and accurate.

Instead, in this post I'm going to explain how I use it to test our roles, with some tips I didn't find in the official documentation. I also played with Gitlab CI so molecule tests are run on each git push, but it will be for another post.

molecule's files

We start by creating a molecule directory in our role:

$ molecule init scenario --verifier-name goss --driver-name docker --role-name <role>

It will create a directory named molecule/ in your role's directory. You can have multiple scenarios for your role, stored under molecule/ directory. By default, molecule init will create the scenario default:

$ tree molecule/
molecule/
└── default
    ├── Dockerfile.j2
    ├── INSTALL.rst
    ├── molecule.yml
    ├── playbook.yml
    ├── tests
    │   └── test_default.yml
    └── verify.yml

Each file above are just here to help you to get started and can be edited as you want. Options we passed to the molecule init command are just here to alter these files. I usually prefer copying the molecule/ directory from an existing role (which is totally acceptable) since I have done some extra customization on it.

molecule.yml

molecule.yml is the only molecule configuration file. Here is the molecule.yml we use in our roles :

---
dependency:
  name: galaxy
driver:
  name: docker
lint:
  name: yamllint
platforms:
  - name: molecule-monitoring-tools
    image: geerlingguy/docker-debian9-ansible:latest
    command: ${MOLECULE_DOCKER_COMMAND:-""}
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
    privileged: true
    pre_build_image: true
provisioner:
  name: ansible
  lint:
    name: ansible-lint
  options:
    skip-tags: molecule-converge-notest
  log: true
scenario:
  name: default

molecule is designed to be very extensible. For instance, it uses the galaxy dependency resolver (dependencies are defined in the meta/main.yml file of your role) and ansible as a provisioner (this is the only one supported for now but we can imagine molecule will be able to use other configuration management tools).

We use yamllint as the linter for all YAML file of the role (including molecule's own files!) and ansible-lint for ansible files.

For some reasons, we don't want to run some Ansible tasks when testing the role. molecule let us pass any ansible-playbook option with the dict provisioner:options. Here we excluded tasks which have the molecule-converge-notest tag on them.

And finally we tell molecule to run the role in a Docker container. molecule supports a large range of drivers, including LXC containers, Vagrant VMs and common public cloud providers. Even if we use LXC containers on production, I choose to use the Docker driver with molecule because, first it's more easy to setup on a personal machine, and second because this is the only supported option if we want to run molecule tests on Gitlab's shared runners (using Docker in Docker), I will talk about it later in a second post.

Most of our roles don't work inside a minimal Docker container, so we use geerlingguy/docker-debian9-ansible Docker image. The image provides a more complete Debian system (with systemd and common tools). It is maintained by Jeff Geerling, who wrote a lot of Ansible roles and uses it to test his own roles.

By default, molecule spawns a container named instance. It's a bad idea since I ended up with conflicts when I tried to launch tests on multiple roles at the same time, each of them was executed in the same container. Thus I edited platforms:name key so that it bears the role's name.

Dockerfile.j2

This file is only here if you want to build your Docker container on each test. You can use any Docker image (defined in platforms:image) and this Dockerfile will install requirements such as python, sudo, bash and ca-certificates package to run Ansible.

Since the image we use already contains those dependencies, we instruct molecule not to build another Docker image (platforms:pre_build_image). this way, we also speed up tests.

INSTALL.rst

INSTALL.rst is an informal file containing instructions to others contributors and requirements to run molecule tests. For instance, we need to have the Docker engine and docker-py installed on our machine.

playbook.yml

After molecule.yml, this is the second most important file. Once molecule has spawned the container, it will run this playbook.

Here is its content:

---
- name: Converge
  hosts: all
  roles:
    - role: <role>

Pretty simple, it applies onto the spun up container. But if your role requires some variables to be set, or dependencies that aren't declared in your role's meta/main.yml file, you can add them here. For instance:

---
- name: Converge
  hosts: all
  roles:
    - role: <other role>
    - role: <role>
  vars:
    foo: "bar"

verify.yml and tests/test_default.yml

Finally, those two files are for unit tests. verify.yml installs Goss (the verifier tool) in the container and run tests it finds in the tests/ directory. Although it's a very interesting subject, I didn't dig too much into unit tests for now.

Running molecule

First, to enable bash completion:

$ eval "$(_MOLECULE_COMPLETE=source molecule)"

Running molecule without argument will show you the list of available sub-commands. I'm not going to explain all of them, and I don't even sure what all these sub-commands do, I still learning about molecule.

Run linting tools on your role:

$ molecule lint

Test your role into a freshly created container (by executing playbook.yml on it):

$ molecule converge

molecule converge will automatically lint your code prior to run it.

At this time you can either login to the container to manually check what has been done:

$ molecule login

Or replay molecule converge if you obtained errors and have fixed them in your role.

molecule converge allows us to pass almost any ansible-playbook option. For instance, you may want to debug tasks with a specific tag:

$ molecule converge -- -vvv --tags <tag>

To destroy the running container so you can start with a new one:

$ molecule destroy

If you just want to start a container but don't execute anything in it (for instance if you want to known the state of a file in a brand new install):

$ molecule prepare

To run idempotence check (molecule will simply run the playbook.yml a second time and check that no tasks report a changed state):

$ molecule idempotence

All these previous commands are useful while you are developing a role. Once you have done or did only minors edits (you are pretty confident that you didn't break anything), it's handy to run all the test suite at once:

$ molecule test

It will lint your code, start a new container, apply playbook.yml, verify idempotence, run unit tests and finally destroy the container.