Infrastructure Testing with Terratest & Kitchen for Terraform Modules

As a senior software engineer and technical writer, I’ve seen firsthand the evolution of infrastructure management. From manual provisioning to configuration management and now to Infrastructure as Code (IaC) with tools like Terraform, the pace of change is relentless. While IaC brings immense benefits in terms of automation, consistency, and repeatability, it also introduces a critical challenge: how do we ensure our infrastructure code is correct, secure, and performs as expected before it ever touches production?

The answer, just like with application code, lies in robust testing. But testing infrastructure is not the same as testing a CRUD API. It involves provisioning real cloud resources, verifying their state, and then tearing them down. This article dives deep into two powerful tools that enable this crucial practice for Terraform modules: Terratest and Kitchen-Terraform. We’ll explore their capabilities, walk through practical examples, discuss their architectural approaches, and help you choose the right tool for your specific testing needs.

The Critical Need for Infrastructure Testing

Imagine deploying an application with a bug. Depending on the severity, it might lead to a crash, incorrect data, or a poor user experience. Now, imagine deploying infrastructure with a bug: a misconfigured firewall rule, an incorrectly sized database, a missing IAM permission, or a subnet with an overlapping CIDR block. The consequences can be far more severe:

• **Downtime and Service Outages**: Incorrect network configurations or resource provisioning can bring down entire applications.
• **Security Vulnerabilities**: Open security groups, misconfigured IAM policies, or publicly exposed resources can lead to data breaches and compliance violations.
• **Cost Overruns**: Over-provisioned resources or forgotten resources can silently rack up significant cloud bills.
• **Inconsistent Environments**: Lack of testing can lead to subtle differences between development, staging, and production environments, causing “works on my machine” syndrome for infrastructure.
• **Reduced Confidence and Slower Innovation**: Teams become hesitant to make changes to infrastructure if they lack confidence in its correctness, slowing down delivery.

Traditional manual testing of infrastructure, often performed by developers or operations teams clicking through cloud consoles, is slow, error-prone, and unsustainable. It doesn’t scale with the complexity and velocity of modern cloud environments. This is why we must shift left, adopting automated testing practices for our IaC, just as we do for our application code.

Levels of Infrastructure Testing

Before diving into Terratest and Kitchen-Terraform, it’s helpful to understand the different levels at which we can test our infrastructure code:

1. Static Analysis/Linting: This is the first line of defense. Tools like terraform validate, tflint, and Checkov examine your Terraform code without deploying it, checking for syntax errors, adherence to best practices, and potential security misconfigurations. While crucial, static analysis can’t verify runtime behavior.

2. Unit Testing (Conceptual for IaC): Unlike application code, true “unit testing” for IaC is challenging because a “unit” often implies a single, isolated resource. For Terraform, this might involve testing a small, self-contained module’s outputs or its ability to render a correct plan, but it typically doesn’t involve actual deployment.

3. Integration Testing: This is where Terratest and Kitchen-Terraform shine. Integration tests involve deploying a small set of real cloud resources defined by your Terraform module(s) and then verifying their interactions, configurations, and outputs. This ensures that individual components work correctly together.

4. End-to-End (E2E) Testing: E2E tests deploy a complete infrastructure stack, often including application deployments on top of it, and then verify the entire system’s functionality from a user’s perspective. These tests are more complex, slower, and more expensive, but they provide the highest confidence in the overall solution.

Our focus here will primarily be on integration and acceptance testing, which typically falls under the umbrella of integration/E2E testing in the context of infrastructure.

<

>

Terratest: Driving Real-World Infrastructure Validation with Go

What is Terratest?

Terratest is an open-source Go library developed by Gruntwork that makes it easier to write automated tests for your infrastructure code. It’s designed to help you provision real infrastructure (e.g., AWS, Azure, GCP, Kubernetes), verify its behavior, and then tear it down. Because it’s written in Go, it leverages Go’s powerful testing framework and ecosystem, making it a natural choice for Go developers or teams already using Go.

Key Features of Terratest

• **Real Infrastructure Provisioning**: Terratest directly interacts with the Terraform CLI to deploy and destroy actual cloud resources. This means your tests run against the same cloud provider APIs and configurations that your production environment will use.
• **Cloud Provider APIs**: It provides helper functions and abstractions for interacting with various cloud provider APIs (AWS SDK, Azure SDK, GCP SDK) to verify resource attributes, state, and behavior.
• **System Utilities**: Terratest includes utilities for common tasks like executing shell commands, making HTTP requests, performing DNS lookups, and even SSHing into instances. This allows for comprehensive verification beyond just Terraform outputs.
• **Idempotent Testing**: Designed for a “deploy, verify, destroy” lifecycle, ensuring that tests are isolated and don’t leave lingering resources.
• **Go Ecosystem**: Being a Go library, it integrates seamlessly with Go’s standard testing package (go test), allowing for parallel test execution, reporting, and familiar assertion patterns.

When to Use Terratest?

• When you need to perform complex integration tests that involve provisioning actual cloud resources across multiple services (e.g., VPC, EC2, RDS, Load Balancers).
• When you want to verify end-to-end scenarios, such as deploying an EC2 instance, installing a web server, and then hitting its public IP to check if it’s serving content.
• When your team is comfortable with Go or already uses Go for other development tasks.
• For testing multi-cloud or hybrid infrastructure setups where direct API interaction is critical.
• To test the full lifecycle of your Terraform modules, including upgrades and downgrades (though the latter is more advanced).

Terratest Architecture (in words)

Conceptually, Terratest acts as an orchestration layer written in Go that wraps around the Terraform CLI and cloud provider SDKs. Here’s how it generally works:

1. Test Code (Go): You write your test logic in Go files (e.g., *_test.go). These files define your test cases, set up test data, and specify the steps for provisioning and verification.

2. Terraform CLI Interaction: Within your Go test, you call Terratest helper functions (e.g., terraform.InitAndApply, terraform.Output, terraform.Destroy). These functions, in turn, execute the actual terraform commands against your Terraform module’s directory.

3. Cloud Provider SDKs: Once Terraform provisions resources, Terratest allows you to use its built-in functions or directly interact with the underlying cloud provider SDKs (e.g., AWS SDK for Go, Azure SDK for Go) to query the state of the deployed resources. For example, you might query an EC2 instance to check its tags, security groups, or IP address.

4. Assertions: Terratest integrates with Go’s standard testing library and assertion libraries like Testify. You use these to assert that the actual state of the resources (obtained from cloud APIs or Terraform outputs) matches your expected state.

5. Teardown (Deferred): Crucially, Terratest tests typically include a defer terraform.Destroy(t, terraformOptions) call. This ensures that all resources provisioned during the test are reliably torn down, even if the test fails, preventing orphaned resources and unexpected costs.

Terratest Code Example: Testing a Basic EC2 Instance

Let’s walk through an example of testing a simple Terraform module that provisions an AWS EC2 instance. We’ll verify that the instance is running and has the correct tags.

First, define your Terraform module. Create a directory structure like this:

├── ec2-module/
│   ├── main.tf
│   ├── variables.tf
│   └── outputs.tf
└── test/
    └── terraform_ec2_test.go

ec2-module/main.tf:

resource "aws_instance" "example" {
  ami           = var.ami_id
  instance_type = var.instance_type
  tags = {
    Name        = var.instance_name
    Environment = var.environment
  }
}

output "instance_id" {
  value = aws_instance.example.id
}

output "instance_public_ip" {
  value = aws_instance.example.public_ip
}

ec2-module/variables.tf:

variable "ami_id" {
  description = "The AMI ID for the EC2 instance"
  type        = string
  default     = "ami-053b0d53c27927903" # Example: Amazon Linux 2 AMI (HVM), SSD Volume Type - us-east-1
}

variable "instance_type" {
  description = "The instance type for the EC2 instance"
  type        = string
  default     = "t2.micro"
}

variable "instance_name" {
  description = "The name tag for the EC2 instance"
  type        = string
  default     = "Terratest-Example-Instance"
}

variable "environment" {
  description = "The environment tag for the EC2 instance"
  type        = string
  default     = "test"
}

ec2-module/outputs.tf:

# Outputs are already defined in main.tf for brevity

Now, the Terratest Go test file: test/terraform_ec2_test.go

package test

import (
	"fmt"
	"testing"
	"time"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/ec2"
	"github.com/gruntwork-io/terratest/modules/aws"
	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/stretchr/testify/assert"
)

func TestTerraformEc2Module(t *testing.T) {
	t.Parallel() // Allows tests to run in parallel

	// Specify the AWS region for the test
	awsRegion := "us-east-1" // Make sure to set AWS_REGION or AWS_DEFAULT_REGION environment variable, or configure your AWS CLI.

	// Define Terraform options
	terraformOptions := &terraform.Options{
		// The path to the Terraform module we want to test.
		TerraformDir: "../ec2-module",
		// Variables to pass to the Terraform module.
		Vars: map[string]interface{}{
			"instance_name": "Terratest-Go-Instance",
			"environment":   "integration-test",
		},
		// By default, Terratest will retry a failed apply. We don't want that for tests.
		MaxRetries: 1,
		TimeBetweenRetries: 5 * time.Second,
	}

	// At the end of the test, run `terraform destroy` to clean up any resources that were created.
	defer terraform.Destroy(t, terraformOptions)

	// Run `terraform init` and `terraform apply`. Fail the test if there are any errors.
	terraform.InitAndApply(t, terraformOptions)

	// Get the instance ID from the Terraform output
	instanceID := terraform.Output(t, terraformOptions, "instance_id")
	assert.NotEmpty(t, instanceID, "Instance ID should not be empty")

	// Verify the instance state using the AWS SDK through Terratest helpers
	// Get the EC2 instance details
	ec2Client := aws.NewEc2Client(t, awsRegion)
	input := &ec2.DescribeInstancesInput{
		InstanceIds: []*string{aws.String(instanceID)},
	}

	result, err := ec2Client.DescribeInstances(input)
	assert.NoError(t, err, "Failed to describe EC2 instance")
	assert.NotEmpty(t, result.Reservations, "No reservations found for instance ID: %s", instanceID)
	assert.NotEmpty(t, result.Reservations[0].Instances, "No instances found in reservation for ID: %s", instanceID)

	instance := result.Reservations[0].Instances[0]

	// Assert instance state
	assert.Equal(t, "running", *instance.State.Name, "Instance should be in 'running' state")

	// Assert tags
	expectedTags := map[string]string{
		"Name":        "Terratest-Go-Instance",
		"Environment": "integration-test",
	}

	actualTags := make(map[string]string)
	for _, tag := range instance.Tags {
		actualTags[*tag.Key] = *tag.Value
	}

	for k, v := range expectedTags {
		assert.Equal(t, v, actualTags[k], fmt.Sprintf("Tag '%s' value mismatch", k))
	}

	// Optionally, verify public IP (if applicable and desired)
	publicIP := terraform.Output(t, terraformOptions, "instance_public_ip")
	if publicIP != "" {
		fmt.Printf("Instance %s has public IP: %s\n", instanceID, publicIP)
		// Further tests could involve making an HTTP request to this IP
	}
}

To run this test:

1. Ensure you have Go installed (go.dev/doc/install).
2. Ensure you have Terraform installed (developer.hashicorp.com/terraform/downloads).
3. Configure your AWS credentials (e.g., via AWS CLI config or environment variables).
4. Navigate to the test/ directory and run: go mod init test (if not already initialized).
5. Run: go mod tidy to download dependencies.
6. Finally, run the test: go test -v

You’ll observe Terraform provisioning the EC2 instance, the Go test verifying its state and tags via the AWS API, and then Terraform destroying the instance. This provides high confidence that your EC2 module behaves as intended.

Kitchen-Terraform: Orchestrating Acceptance Tests with InSpec

What is Kitchen-Terraform?

Test Kitchen is a well-established Ruby-based test harness for developing and testing infrastructure code (originally for Chef cookbooks, but now extensible). Kitchen-Terraform is a Test Kitchen provisioner that allows you to use Terraform to provision test infrastructure. It then integrates with a verifier, most commonly InSpec, to run compliance and security tests against the provisioned resources.

Key Features of Kitchen-Terraform

• **Test Kitchen Ecosystem**: Leverages the robust Test Kitchen framework for lifecycle management (create, converge, verify, destroy) and platform abstraction (Docker, Vagrant, various cloud drivers).
• **InSpec Verification**: Integrates seamlessly with InSpec, a human-readable, domain-specific language (DSL) for writing compliance and security tests. This makes tests easy to understand and maintain for non-developers too.
• **Platform Agnostic**: Through Test Kitchen drivers, you can test your Terraform modules against various target platforms, from local Docker containers and Vagrant VMs to actual cloud instances (EC2, Azure, GCP).
• **Acceptance Testing Focus**: Excellent for acceptance testing of modules that deploy operating systems, install software, or configure services within VMs or containers.
• **Ruby-based**: If your team is already familiar with Ruby or Test Kitchen/InSpec for configuration management, this provides a natural extension for Terraform testing.

When to Use Kitchen-Terraform?

• When you need to test Terraform modules that provision virtual machines or containers and then apply configuration (e.g., using user_data, remote-exec provisioners, or a subsequent configuration management tool).
• When you want to verify the internal state of an operating system, installed packages, running services, or file contents.
• When your team is already invested in the Ruby/Test Kitchen/InSpec ecosystem.
• For acceptance testing of application servers, database instances, or other compute resources where guest OS configuration is paramount.
• When you need to manage test environments across different platforms (e.g., test locally with Docker, then in AWS EC2).

Kitchen-Terraform Architecture (in words)

Test Kitchen, with the Kitchen-Terraform plugin, provides a structured approach to testing. Here’s the typical workflow:

1. Configuration (.kitchen.yml): You define your test matrix in a .kitchen.yml file. This YAML file specifies:

   • driver: How Test Kitchen creates the test instance (e.g., kitchen-docker, kitchen-ec2).
   • provisioner: How the instance is configured. For us, this is terraform. It points to your Terraform module’s directory.
   • verifier: How the instance’s state is checked. Typically inspec, pointing to an InSpec profile.
   • platforms: The target operating systems or environments for testing.
   • suites: Different test scenarios, each potentially using different Terraform variables.

2. Create Phase: kitchen create <suite-name>. The driver creates a base test instance (e.g., a Docker container or an EC2 instance) based on the specified platform.

3. Converge Phase: kitchen converge <suite-name>. The Kitchen-Terraform provisioner takes over. It executes terraform init and terraform apply in your module’s directory, provisioning the specified infrastructure. This might include the instance created in the “create” phase, or other cloud resources related to it (e.g., a VPC, security groups).

4. Verify Phase: kitchen verify <suite-name>. The verifier (InSpec) then runs. InSpec connects to the test instance (often via SSH or WinRM, or directly interacts with cloud APIs via plugins) and executes the tests defined in its profile. It checks for installed packages, running services, file contents, network accessibility, etc.

5. Destroy Phase: kitchen destroy <suite-name>. After verification, Test Kitchen ensures all resources provisioned by Terraform and the test instance itself are torn down, cleaning up the environment.

The entire lifecycle can be run with a single command: kitchen test <suite-name>.

<

>

Kitchen-Terraform Code Example: Testing an Nginx Web Server

Let’s create a Terraform module that provisions a local Docker container (using the Docker provider) and installs Nginx via a local-exec provisioner. We’ll then use Kitchen-Terraform with InSpec to verify Nginx is running and serving content.

Directory structure:

├── nginx-webserver/
│   ├── main.tf
│   ├── versions.tf
│   └── .kitchen.yml
└── test/
    └── integration/
        └── default/
            └── inspec/
                └── nginx_spec.rb

nginx-webserver/versions.tf (to define Docker provider):

terraform {
  required_providers {
    docker = {
      source  = "kreuzwerker/docker"
      version = "~> 2.15"
    }
  }
}

nginx-webserver/main.tf:

resource "docker_image" "nginx" {
  name         = "nginx:latest"
  keep_locally = true
}

resource "docker_container" "nginx" {
  image = docker_image.nginx.name
  name  = "test-nginx-container"
  ports {
    internal = 80
    external = 8080
  }

  // Use local-exec to run commands *inside* the container
  // Note: This is a simplified example. In a real scenario, you'd provision an EC2/VM and
  // connect to it via SSH or remote-exec, or use a provisioner like Ansible/Chef.
  // For a Docker container used by Kitchen, InSpec will connect directly.
  // This local-exec here is illustrative of *how* a module might configure a server.
  // In a kitchen-terraform test for a VM, you'd have user_data to install nginx.
  // For a Docker container with Kitchen-Terraform, the 'converge' phase will just run `docker run`.
  // The 'verify' phase (InSpec) will then connect to the running container directly.
  // So, for this example, let's assume nginx is part of the image, or we configure it
  // in a more realistic way for Kitchen-Terraform/InSpec, e.g., via user_data for a cloud VM.
  // For a Docker container, the image itself would contain Nginx.
  // Let's adjust to be more realistic for Kitchen+Terraform.
  // We'll provision a Docker container and assume InSpec can connect directly to verify.
}

output "nginx_port" {
  value = docker_container.nginx.ports[0].external
}

Correction on `main.tf` explanation for Docker: For a Docker container, you’d typically use an image that already has Nginx or pass a `command` or `entrypoint` to install it. Terraform’s local-exec runs on the host, not inside the container. In Kitchen, InSpec connects to the running container. So, our `main.tf` will just launch an Nginx container. InSpec will then verify it.

nginx-webserver/.kitchen.yml:

---
driver:
  name: docker # Use the Docker driver for local testing

provisioner:
  name: terraform
  # Location of your Terraform module relative to .kitchen.yml
  terraform_directory: .

verifier:
  name: inspec

platforms:
  - name: docker # Our platform is a Docker container

suites:
  - name: default
    # You can pass variables to your Terraform module here
    # variables:
    #   some_var: "some_value"
    verifier:
      # Location of your InSpec profile relative to .kitchen.yml
      inspec_directory: test/integration/default/inspec

test/integration/default/inspec/nginx_spec.rb:

describe docker_container('test-nginx-container') do

  it { should be_running }

  its('image') { should eq 'nginx:latest' }

  its('repo') { should eq 'nginx' }

  its