Test driven infrastructure development

Your race director for the next hour

Felix Peters

Operations Core Tooling

DevOps & Operations Engineer

Github · flxpeters.de

Let’s talk about

Infrastructure as Code


itΒ΄s just really cool to describe your whole infrastructure with code


  • Versioned and reproducible
  • Speedup time to market
  • Extracted from those head monopolies



Code without tests is broken by design.

Jacob Kaplan-Moss

Why testing is important

  1. Proof that the code works
  2. Prevent errors in production
  3. Increase code quality and structure
  4. Express the functionality in human expectation rather than code and define the scope of a project

Example Project

This is not testable!

So, how do we build this test driven?


Divide and conquer

  • Split the project in testable modules
  • Test each modules in isolation
  • Integrate modules and test them together

IaC test pyramid

Part 1

  • Dockerized Python App
    Package App as Docker image
  • Ansible role Docker
    Install Docker with Ansible
  • Ansible role Application
    Setup App with Docker

Part 2

  • Packer
    Package App and Docker to an AMI with Packer
  • Terraform Module
    Setup EC2 instances

Part 1.1

Build and test a Docker image


    FROM python:alpine3.11

    RUN adduser --disabled-password --gecos '' -u 1001 app
    USER app
    WORKDIR /home/app

    COPY --chown=app:app app.py app.py

    EXPOSE 8000

    CMD [ "python3", "app.py" ]

Container Structure Tests

Static analysis Github

Validate the structure of a container image

schemaVersion: 2.0.0

- name: "app.py"
    path: "/home/app/app.py"
    shouldExist: true
    uid: 1001

exposedPorts: ["8000"]
cmd: ["python3", "app.py"]
workdir: "/home/app"


Linter Static analysis Github

A smarter Dockerfile linter that helps you build best practice Docker images

Curl & Bash

Unit test

docker run --rm --name tdd-example-test -d -p 8000:8000 tdd-example:latest

until $(curl --output /dev/null --silent --fail http://localhost:8000); do
    if [ ${attempt_counter} -eq ${max_attempts} ];then
    echo "Max attempts reached"
    exit 1
    echo '.'
    sleep 5

docker stop tdd-example-test


  • Testing Docker images is possible, but rarely done
  • Enforce best practice by using a linter
  • Uses familiar tools to verify the behavior of images

Part 1.2

Testing Ansible roles


Test Framework Github Docs

Modular framework for testing Ansible roles in many scenarios and distributions

Ansible, Docker, Podman, Vagrant, Cloud providers

Ansible-Lint, yamllint, Ansible, Inspec, Testinfra, Goss


  1. Install dependencies via Ansible Galaxy
  2. Lint the Ansible role code
  3. Prepare: Start one ore more test instances using a driver and apply prepare steps
  4. Converge: Apply the role via an Ansible playbook
  5. Run the playbook again to ensure idempotence
  6. Verify: Run one or more verifiers
  7. Cleanup

Ansible Lint

Linter Github

Checks playbooks for practices and behaviour that could potentially be improved.

$ ansible-lint geerlingguy.apache

[502] All tasks should be named
Task/Handler: include_vars apache-22.yml


Linter Github Example

A linter for YAML files.
Enforce best practices and formats for yaml files.

Ansible as verifier

Unit test Integration test Github

Use ansible facts and modules and assert as test driver.

- name: Populate service facts

- name: Assert services are running
      - ansible_facts.services['docker.service'].state == 'running'

- name: Check Cadvisor web is available
  uri: url="http://localhost:8089/containers/" status_code=[200]
  register: result
  until: result.status == 200
  retries: 60
  delay: 1


Unit test Github Docs

With Testinfra you can write unit tests in Python to test actual state of your servers.

def test_nginx_is_installed(host):
    nginx = host.package("nginx")
    assert nginx.is_installed
    assert nginx.version.startswith("1.2")

def test_nginx_running_and_enabled(host):
    nginx = host.service("nginx")
    assert nginx.is_running
    assert nginx.is_enabled

Make use of the well known pytest framework. Inspired by Serverspec.


Unit test Github

Goss is a simple YAML based tool for validating a server’s configuration. Allows generating tests from current system status. No coding required.

$ cat goss.yaml
        listening: true
        enabled: true
        running: true

Can also be used for verifying docker images!

Chef Inspec

Unit test Integration test Github Docs

Chef InSpec is an testing framework for infrastructure with a human- and machine-readable DSL.

describe port(80) do
    it { should_not be_listening }

describe port(443) do
    it { should be_listening }
    its('protocols') {should include 'tcp'}

Provides a wide range of controls for servers and cloud providers like AWS or Azure.

Molecule conclusion

  • Testing Ansible roles is a solved problem
  • Use molecule as Ansible’s native testing framework
  • Use one ore more linter
  • Use the verifier you like best
  • Testing in docker is fast but can be complicated

Part 2.1

Build an test AMI with Packer


  1. Create a EC2 instance
  2. Apply Ansible roles to this instance
  3. Test the result
  4. Convert to AMI


Build tool Website Github

Packer is a free and open source tool for creating golden images for multiple platforms

Supports multiple platforms like AWS, Azure, GCP, VMWare and Docker.

Chef Inspec

Unit test Integration test Github Docs

Chef InSpec is an testing framework for infrastructure with a human- and machine-readable DSL.

describe port(80) do
    it { should_not be_listening }

describe port(443) do
    it { should be_listening }
    its('protocols') {should include 'tcp'}

Provides a wide range of controls for servers and cloud providers like AWS or Azure.


πŸ’‘ Tips

  • Add a Docker builder for local testing of Ansible playbook integration
  • Packer can also be used to build Docker images
  • Packer can build multi cloud images by using multiple builders
  • Use the “Breakpoint Provisioner” for debugging


  • Packer is the defacto standard tool for building golden images
  • Packer integrates best with with Inspec

Part 2.2

Test Terraform modules


  1. Code
  2. Deploy infrastructure
  3. Validate it works
  4. Undeploy the infrastructure
  5. Repeat


Testing infrastructure requires a real world infrastructure!

⚠ Warning

This will deploy and destroy many resources!

Tipp: Use an isolated sandbox account and nuke this account every night

βš’ Tools


Test framework Github

Kitchen-Terraform enables verification of infrastructure systems provisioned with Terraform.

  • Written in Ruby, configured via Yaml
  • Use the well known Test Kitchen from Chef the ecosystem
  • Same approach for converge and verify development cycle like Molecule
  • Integarates with Inspec


Test framework Unit test Integration test Github

Terratest is a Go library that makes it easier to write automated tests for your infrastructure code. It provides a variety of helper functions and patterns for common infrastructure testing tasks.

  • Support for Terraform, Packer, Kubernetes and Docker
  • Validate using the Golang testing framework
  • Comes with handy helper functions for e.g. HTTP requests
  • Verify cloud resources by using the GO SDK functions

Examples: https://github.com/gruntwork-io/terratest/



  • πŸ‘ Good integration with Inspec
  • πŸ‘ Support for complex setups and tests
  • πŸ‘Ž Complex Configuration
  • πŸ‘Ž Ruby dependency management


  • πŸ‘ Easy to learn (when you know Go)
  • πŸ‘ Support for multiple tools
  • πŸ‘ Full power of Golang testing framework
  • πŸ‘Ž No integration for Inspec and other test tools

πŸ’‘ Tips

  • Use namespaced tests to avoid naming conflicts
  • Use a sandbox account to reduce blast radius
  • Build combined tests to test modules together

Further prospects

  • Regula: Convert Terraform Plan to Json and apply Rego policy rules to enforce infrastructure policies
  • Open Policy Agent: Calculate blast radius on Terraform plan file to enforce infrastructure policies
  • Terraform Sentinal: Terraform embedded policy enforcement engine

πŸ’‘ Key takeaways

  • Testing infrastructure code is a solved problem!

    • Evaluate and learn the tools and methods
    • Testing will give you confidence to change things!
    • Test as early and often as possible
    • Always lint and use static code analytics
    • Add unit and integration tests whenever possible
  • Split large projects into handy modules

    • If a unit is to big for testing ➑ refactor it!
    • Build and test modules in isolation
    • Use tests to describe whats the purpose of a module