Let’s talk about

Infrastructure as Code

Why?

it´s just really cool to describe your whole infrastructure with code

Jonas

• Versioned and reproducible
• Speedup time to market
• Extracted from those head monopolies

Expectation

Reality

Code without tests is broken by design.

Jacob Kaplan-Moss

Why testing is important

1. Proof that the code works
2. Prevent errors in production
3. Increase code quality and structure
4. Express the functionality in human expectation rather than code and define the scope of a project

Example Project

So, how do we build this test driven?

Answer:

Divide and conquer

• Split the project in testable modules
• Test each modules in isolation
• Integrate modules and test them together

Part 1.1

Build and test a Docker image

Dockerfile

    FROM python:alpine3.11

    RUN adduser --disabled-password --gecos '' -u 1001 app
    USER app
    WORKDIR /home/app

    COPY --chown=app:app app.py app.py

    EXPOSE 8000

    CMD [ "python3", "app.py" ]

Container Structure Tests

Static analysis Github

Validate the structure of a container image

schemaVersion: 2.0.0

fileExistenceTests:
- name: "app.py"
    path: "/home/app/app.py"
    shouldExist: true
    uid: 1001

metadataTest:
exposedPorts: ["8000"]
cmd: ["python3", "app.py"]
workdir: "/home/app"

Hadolint

Linter Static analysis Github

A smarter Dockerfile linter that helps you build best practice Docker images

Curl & Bash

Unit test

docker run --rm --name tdd-example-test -d -p 8000:8000 tdd-example:latest

attempt_counter=0
max_attempts=5
until $(curl --output /dev/null --silent --fail http://localhost:8000); do
    if [ ${attempt_counter} -eq ${max_attempts} ];then
    echo "Max attempts reached"
    exit 1
    fi
    echo '.'
    attempt_counter=$(($attempt_counter+1))
    sleep 5
done

docker stop tdd-example-test

Conclusion

• Testing Docker images is possible, but rarely done
• Enforce best practice by using a linter
• Uses familiar tools to verify the behavior of images

Part 1.2

Testing Ansible roles

Molecule

Test Framework Github Docs

Modular framework for testing Ansible roles in many scenarios and distributions

Drivers:
Ansible, Docker, Podman, Vagrant, Cloud providers

Verifiers:
Ansible-Lint, yamllint, Ansible, Inspec, Testinfra, Goss

Workflow

1. Install dependencies via Ansible Galaxy
2. Lint the Ansible role code
3. Prepare: Start one ore more test instances using a driver and apply prepare steps
4. Converge: Apply the role via an Ansible playbook
5. Run the playbook again to ensure idempotence
6. Verify: Run one or more verifiers
7. Cleanup

Ansible Lint

Linter Github

Checks playbooks for practices and behaviour that could potentially be improved.

$ ansible-lint geerlingguy.apache

[502] All tasks should be named
/Users/chouseknecht/.ansible/roles/geerlingguy.apache/tasks/main.yml:29
Task/Handler: include_vars apache-22.yml

yamllint

Linter Github Example

A linter for YAML files.
Enforce best practices and formats for yaml files.

Ansible as verifier

Unit test Integration test Github

Use ansible facts and modules and assert as test driver.

- name: Populate service facts
  service_facts:

- name: Assert services are running
  assert:
    that:
      - ansible_facts.services['docker.service'].state == 'running'

- name: Check Cadvisor web is available
  uri: url="http://localhost:8089/containers/" status_code=[200]
  register: result
  until: result.status == 200
  retries: 60
  delay: 1

Testinfra

Unit test Github Docs

With Testinfra you can write unit tests in Python to test actual state of your servers.

def test_nginx_is_installed(host):
    nginx = host.package("nginx")
    assert nginx.is_installed
    assert nginx.version.startswith("1.2")

def test_nginx_running_and_enabled(host):
    nginx = host.service("nginx")
    assert nginx.is_running
    assert nginx.is_enabled

Make use of the well known pytest framework. Inspired by Serverspec.

Goss

Unit test Github

Goss is a simple YAML based tool for validating a server’s configuration. Allows generating tests from current system status. No coding required.

$ cat goss.yaml
port:
    tcp:22:
        listening: true
        ip:
        - 0.0.0.0
service:
    sshd:
        enabled: true
        running: true

Can also be used for verifying docker images!

Chef Inspec

Unit test Integration test Github Docs

Chef InSpec is an testing framework for infrastructure with a human- and machine-readable DSL.

describe port(80) do
    it { should_not be_listening }
end

describe port(443) do
    it { should be_listening }
    its('protocols') {should include 'tcp'}
end

Provides a wide range of controls for servers and cloud providers like AWS or Azure.

Molecule conclusion

• Testing Ansible roles is a solved problem
• Use molecule as Ansible’s native testing framework
• Use one ore more linter
• Use the verifier you like best
• Testing in docker is fast but can be complicated

Part 2.1

Build an test AMI with Packer

Workflow

1. Create a EC2 instance
2. Apply Ansible roles to this instance
3. Test the result
4. Convert to AMI

Packer

Build tool Website Github

Packer is a free and open source tool for creating golden images for multiple platforms

Supports multiple platforms like AWS, Azure, GCP, VMWare and Docker.

Chef Inspec

Unit test Integration test Github Docs

Chef InSpec is an testing framework for infrastructure with a human- and machine-readable DSL.

describe port(80) do
    it { should_not be_listening }
end

describe port(443) do
    it { should be_listening }
    its('protocols') {should include 'tcp'}
end

Provides a wide range of controls for servers and cloud providers like AWS or Azure.

https://asciinema.org/a/B2Uk82Myi4FbQSKmQ73cr5PGk

💡 Tips

• Add a Docker builder for local testing of Ansible playbook integration
• Packer can also be used to build Docker images
• Packer can build multi cloud images by using multiple builders
• Use the “Breakpoint Provisioner” for debugging

Conclusion

• Packer is the defacto standard tool for building golden images
• Packer integrates best with with Inspec

Part 2.2

Test Terraform modules

Workflow

1. Code
2. Deploy infrastructure
3. Validate it works
4. Undeploy the infrastructure
5. Repeat

Fact

Testing infrastructure requires a real world infrastructure!

⚠ Warning

This will deploy and destroy many resources!

Tipp: Use an isolated sandbox account and nuke this account every night

⚒ Tools

Kitchen-Terraform

Test framework Github

Kitchen-Terraform enables verification of infrastructure systems provisioned with Terraform.

• Written in Ruby, configured via Yaml
• Use the well known Test Kitchen from Chef the ecosystem
• Same approach for converge and verify development cycle like Molecule
• Integarates with Inspec

Terratest

Test framework Unit test Integration test Github

Terratest is a Go library that makes it easier to write automated tests for your infrastructure code. It provides a variety of helper functions and patterns for common infrastructure testing tasks.

• Support for Terraform, Packer, Kubernetes and Docker
• Validate using the Golang testing framework
• Comes with handy helper functions for e.g. HTTP requests
• Verify cloud resources by using the GO SDK functions

Examples: https://github.com/gruntwork-io/terratest/

Comparison

Kitchen-Terraform

• 👍 Good integration with Inspec
• 👍 Support for complex setups and tests
• 👎 Complex Configuration
• 👎 Ruby dependency management

Terratest

• 👍 Easy to learn (when you know Go)
• 👍 Support for multiple tools
• 👍 Full power of Golang testing framework
• 👎 No integration for Inspec and other test tools

💡 Tips

• Use namespaced tests to avoid naming conflicts
• Use a sandbox account to reduce blast radius
• Build combined tests to test modules together

Further prospects

• Regula: Convert Terraform Plan to Json and apply Rego policy rules to enforce infrastructure policies
• Open Policy Agent: Calculate blast radius on Terraform plan file to enforce infrastructure policies
• Terraform Sentinal: Terraform embedded policy enforcement engine

💡 Key takeaways

• Testing infrastructure code is a solved problem!

  • Evaluate and learn the tools and methods
  • Testing will give you confidence to change things!
  • Test as early and often as possible
  • Always lint and use static code analytics
  • Add unit and integration tests whenever possible

• Split large projects into handy modules

  • If a unit is to big for testing ➡ refactor it!
  • Build and test modules in isolation
  • Use tests to describe whats the purpose of a module

Questions?