Infrastructure Testing with Terratest and Kitchen for Terraform Modules

As a senior software engineer at Khadervali.com, I’ve seen firsthand how the landscape of infrastructure management has evolved. Gone are the days of manual server provisioning and ad-hoc configuration. Infrastructure as Code (IaC) has revolutionized how we build, deploy, and manage our cloud resources, bringing software engineering principles like version control, automation, and modularity to the infrastructure layer. Terraform, in particular, has emerged as a dominant player, allowing us to define our desired state declaratively across various cloud providers.

However, simply writing Terraform code isn’t enough. Just like any other codebase, our infrastructure code is prone to bugs, misconfigurations, and unintended side effects. An untested Terraform module can lead to production outages, security vulnerabilities, and significant operational overhead. This is where robust infrastructure testing becomes paramount.

In this comprehensive article, we’ll dive deep into two powerful tools that enable us to build confidence in our Terraform modules: Terratest and Test Kitchen. We’ll explore their unique strengths, how they integrate with Terraform, and how to leverage them to create resilient, reliable, and thoroughly tested infrastructure.

The Imperative of Infrastructure Testing

Before we delve into the tools, let’s solidify *why* infrastructure testing is non-negotiable in modern DevOps practices.

The Challenges of Untested Infrastructure

• Configuration Drift: Manual changes or flawed automation can lead to discrepancies between your defined IaC and the actual infrastructure state, making debugging a nightmare.
• Reliability Issues: Untested modules can deploy resources incorrectly, leading to application downtime or degraded performance.
• Security Vulnerabilities: Misconfigured security groups, IAM policies, or network rules can expose your infrastructure to unauthorized access.
• Cost Overruns: Incorrectly provisioned resources (e.g., wrong instance types, unneeded services) can lead to unexpected cloud bills.
• Slow Feedback Loop: Without automated tests, you might only discover issues in production, leading to costly and stressful rollbacks.
• Refactoring Risk: Changing a Terraform module without comprehensive tests makes it difficult to ensure that existing functionality remains intact.

Automated testing addresses these challenges by providing a fast, repeatable, and reliable way to validate your infrastructure code before it impacts production environments.

What Constitutes “Infrastructure Testing”?

Infrastructure testing, in the context of IaC, typically involves several layers:

1. Static Analysis/Linting: Checking code for syntax errors, best practices, and potential security issues without deploying anything (e.g., terraform validate, TFLint, Checkov).
2. Unit Testing: Verifying individual components or modules in isolation. For Terraform, this often means ensuring resources are correctly defined with expected properties.
3. Integration Testing: Testing how multiple modules or infrastructure components interact with each other in a deployed environment. This might involve deploying a small stack and verifying connectivity or data flow.
4. End-to-End Testing: Deploying a complete application stack and verifying that it functions as expected from a user’s perspective, including application-level health checks.

Terratest and Test Kitchen primarily focus on unit and integration testing of deployed infrastructure, making them critical tools in your IaC testing arsenal.

Terratest: Go-Based Testing for Real Infrastructure

What is Terratest?

Terratest is an open-source Go library developed by Gruntwork. It’s designed to help you write automated tests for your infrastructure code, regardless of whether it’s written in Terraform, Packer, Docker, or other tools. The core idea behind Terratest is to:

1. Deploy: Spin up real infrastructure in a real cloud environment (AWS, Azure, GCP, Kubernetes, etc.) using your IaC.
2. Validate: Make assertions against that deployed infrastructure. This could involve checking resource attributes, connectivity, or even running commands on provisioned instances.
3. Destroy: Tear down the infrastructure to clean up resources and avoid incurring unnecessary costs.

By interacting with actual cloud provider APIs, Terratest ensures that your Terraform modules not only synthesize valid plans but also *actually* provision the resources as expected.

Why Go for Infrastructure Testing?

Go is an excellent choice for Terratest for several reasons:

• Performance: Go is fast and compiles to a single binary, making tests quick to execute.
• Concurrency: Go’s goroutines and channels make it easy to write concurrent tests, useful for deploying and validating multiple resources in parallel.
• Rich Ecosystem: Go has robust libraries for interacting with cloud provider APIs, HTTP, SSH, and more.
• Testing Framework: Go has a built-in testing package (testing) that is simple yet powerful.

Setting Up Your Terratest Environment

To use Terratest, you’ll need:

1. Go: Install Go (version 1.16 or higher is recommended).
2. Cloud Provider CLI: Ensure you have the necessary CLI tools configured and authenticated for your cloud provider (e.g., AWS CLI, Azure CLI, gcloud CLI).
3. Terraform: Make sure Terraform is installed and available in your PATH.

A typical Terratest project structure looks like this:

my-terraform-module/
├── main.tf
├── variables.tf
├── outputs.tf
└── test/
    └── my_module_test.go

Core Concepts: Deploy, Validate, Destroy

Let’s break down the fundamental flow of a Terratest test:

package test

import (
	"testing"
	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/stretchr/testify/assert" // A popular assertion library for Go
)

func TestMySimpleTerraformModule(t *testing.T) {
	t.Parallel() // Allows tests to run in parallel

	// Construct the terraform options with default retryable errors to handle the occasional
	// API issues that affect some cloud providers.
	terraformOptions := terraform.With     DefaultRetryableErrors(t, &terraform.Options{
		TerraformDir: "../", // Path to the Terraform module we want to test
		// Variables to pass to our Terraform module
		Vars: map[string]interface{}{
			"example_name": "my-test-resource",
			"region":       "us-east-1", // Example region
		},
	})

	// At the end of the test, run `terraform destroy` to clean up any resources that were created.
	defer terraform.Destroy(t, terraformOptions)

	// This will run `terraform init` and `terraform apply`.
	terraform.InitAndApply(t, terraformOptions)

	// Now, validate the deployed infrastructure.
	// We can fetch outputs from the Terraform module.
	outputValue := terraform.Output(t, terraformOptions, "my_output")

	// Make assertions about the output.
	assert.Equal(t, "expected-value", outputValue, "Output value should match expected.")

	// You can also interact with the cloud provider APIs directly here
	// to verify resource properties. For example, for AWS:
	// awsRegion := terraformOptions.Vars["region"].(string)
	// actualBucketName := outputValue // Assuming outputValue is an S3 bucket name
	// bucketExists := aws.IsS3BucketExists(t, awsRegion, actualBucketName)
	// assert.True(t, bucketExists, "S3 bucket should exist")
}

Real-World Scenario: Testing an S3 Bucket Module

Let’s create a simple Terraform module for an S3 bucket and then write a Terratest to verify its creation and properties.

Terraform Module: s3-module/main.tf

resource "aws_s3_bucket" "example" {
  bucket = var.bucket_name
  acl    = var.acl

  tags = {
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

output "bucket_id" {
  value       = aws_s3_bucket.example.id
  description = "The name of the S3 bucket."
}

output "bucket_arn" {
  value       = aws_s3_bucket.example.arn
  description = "The ARN of the S3 bucket."
}

Terraform Module: s3-module/variables.tf

variable "bucket_name" {
  description = "The name of the S3 bucket."
  type        = string
}

variable "acl" {
  description = "The canned ACL to apply. We recommend using 'private' or 'log-delivery'."
  type        = string
  default     = "private"
}

variable "environment" {
  description = "The environment tag for the bucket."
  type        = string
  default     = "test"
}

variable "aws_region" {
  description = "The AWS region to deploy resources into."
  type        = string
  default     = "us-east-1"
}

Terratest: s3-module/test/s3_test.go

package test

import (
	"fmt"
	"testing"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/service/s3"
	"github.com/gruntwork-io/terratest/modules/aws"
	"github.com/gruntwork-io/terratest/modules/random"
	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/stretchr/testify/assert"
)

func TestS3BucketModule(t *testing.T) {
	t.Parallel()

	// Generate a unique bucket name to avoid conflicts
	uniqueID := random.UniqueId()
	bucketName := fmt.Sprintf("khadervali-test-bucket-%s", uniqueID)
	awsRegion := "us-east-1" // Ensure this region is configured in your AWS CLI

	terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
		TerraformDir: "../", // Path to the Terraform S3 module
		Vars: map[string]interface{}{
			"bucket_name": bucketName,
			"acl":         "private",
			"environment": "dev",
			"aws_region":  awsRegion,
		},
		EnvVars: map[string]string{
			"AWS_DEFAULT_REGION": awsRegion, // Set region for Terraform execution
		},
	})

	defer terraform.Destroy(t, terraformOptions) // Clean up resources at the end

	terraform.InitAndApply(t, terraformOptions) // Deploy the S3 bucket

	// Validate outputs
	actualBucketId := terraform.Output(t, terraformOptions, "bucket_id")
	actualBucketArn := terraform.Output(t, terraformOptions, "bucket_arn")

	assert.Equal(t, bucketName, actualBucketId, "Expected bucket ID to match generated name.")
	assert.Contains(t, actualBucketArn, bucketName, "Expected ARN to contain bucket name.")

	// Validate the S3 bucket actually exists and has correct tags using AWS SDK
	s3Client := aws.NewS3Client(t, awsRegion)
	_, err := s3Client.HeadBucket(&s3.HeadBucketInput{Bucket: aws.String(bucketName)})
	assert.NoError(t, err, "S3 bucket should exist.")

	// Get bucket tags
	tags, err := aws.GetS3BucketTags(t, awsRegion, bucketName)
	assert.NoError(t, err, "Should be able to get S3 bucket tags.")
	assert.Equal(t, "dev", tags["Environment"], "Environment tag should be 'dev'.")
	assert.Equal(t, "Terraform", tags["ManagedBy"], "ManagedBy tag should be 'Terraform'.")

	// Validate ACL (requires specific permissions, can be complex to test directly via SDK)
	// For simplicity, we'll assume if it's created, the ACL applied correctly based on the module.
	// More advanced tests might involve creating objects and testing access.
}

To run this test, navigate to the `s3-module/test` directory and execute: go test -v

Advanced Terratest Techniques

• Retry Logic: Terratest includes built-in retry logic (terraform.WithDefaultRetryableErrors) to handle transient cloud API errors, making your tests more robust.
• Parallel Execution: Use t.Parallel() at the beginning of your test functions to run tests concurrently, speeding up your test suite.
• External Dependencies: Terratest allows you to execute shell commands, SSH into instances, and interact with Kubernetes clusters, enabling comprehensive validation.
• Helper Functions: Organize common validation logic into helper functions to promote reusability and maintainability.
• Destroy Confirmation: For critical or expensive resources, Terratest can be configured to require manual confirmation before destruction in CI environments.

Terratest is incredibly versatile, allowing you to validate virtually any aspect of your deployed infrastructure. Its Go-native approach means you have the full power of the Go ecosystem at your fingertips for sophisticated assertions and interactions.

Test Kitchen with kitchen-terraform: Provisioning & Verifying VM/Container Configuration

What is Test Kitchen?

Test Kitchen (often just “Kitchen”) is an integration testing framework developed by Chef. While originally designed for testing Chef cookbooks, its extensible plugin architecture makes it incredibly useful for testing any type of infrastructure configuration, including Terraform modules. Kitchen’s strength lies in its ability to:

1. Provision: Create isolated test environments (e.g., EC2 instances, Docker containers, local VMs) on various platforms.
2. Converge: Apply your configuration management code (or in our case, Terraform) to these provisioned environments.
3. Verify: Run tests against the converged environment to ensure it’s configured as expected, using a verifier like InSpec.
4. Destroy: Tear down the test environment, similar to Terratest.

Why Test Kitchen for Terraform? The kitchen-terraform Driver

While Terratest directly interacts with Terraform and cloud APIs, Test Kitchen, through the kitchen-terraform driver, focuses more on the *outcome* of the Terraform deployment, especially when that outcome involves a virtual machine, container, or network that needs further configuration or interaction. It excels at testing scenarios where you deploy an instance and then need to verify its OS-level configuration, installed software, or network accessibility.

kitchen-terraform acts as a Kitchen driver that understands how to execute Terraform commands (init, apply, output, destroy) and map them to Kitchen’s lifecycle stages.

Setting Up Your Test Kitchen Environment

You’ll need:

1. Ruby: Test Kitchen is a Ruby gem, so you need a Ruby environment (use RVM or rbenv for managing Ruby versions).
2. Bundler: A Ruby gem dependency manager.
3. Terraform: Installed and in your PATH.
4. Cloud Provider CLI: Configured and authenticated.

A typical Kitchen project structure for a Terraform module:

my-ec2-module/
├── main.tf
├── variables.tf
├── outputs.tf
├── Gemfile
├── .kitchen.yml
└── test/
    └── integration/
        └── default/
            └── default_spec.rb # InSpec test file

Core Concepts in .kitchen.yml

The .kitchen.yml file is the heart of Test Kitchen. It defines how your tests will run:

• Driver: Specifies how Kitchen provisions the test environment. For Terraform, this will be kitchen-terraform.
• Provisioner: Specifies how to apply configuration. For kitchen-terraform, this is implicitly Terraform itself.
• Verifier: Specifies the testing framework to use for assertions. InSpec is the de-facto standard for Kitchen.
• Platforms: Defines the target environments where your tests will run (e.g., AWS, Azure, GCP).
• Suites: Defines specific test scenarios, allowing you to test different configurations of your module.

Real-World Scenario: Testing an EC2 Instance Module with InSpec

Let’s create a Terraform module that deploys a simple EC2 instance and then use Test Kitchen with InSpec to verify its properties and basic configuration.

Terraform Module: ec2-module/main.tf

resource "aws_security_group" "ssh_sg" {
  name_prefix = "ssh-sg-"
  description = "Allow SSH inbound traffic"
  vpc_id      = var.vpc_id

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name        = "${var.name_prefix}-ssh-sg"
    Environment = var.environment
  }
}

resource "aws_instance" "example" {
  ami           = var.ami_id
  instance_type = var.instance_type
  key_name      = var.key_pair_name
  vpc_security_group_ids = [aws_security_group.ssh_sg.id]
  subnet_id     = var.subnet_id # Assuming a default VPC or provided subnet

  user_data = <<-EOF
              #!/bin/bash
              echo "Hello from Terraform" > /tmp/hello.txt
              sudo apt-get update
              sudo apt-get install -y nginx
              sudo systemctl enable nginx
              sudo systemctl start nginx
              EOF

  tags = {
    Name        = "${var.name_prefix}-instance"
    Environment = var.environment
  }
}

output "instance_public_ip" {
  value       = aws_instance.example.public_ip
  description = "The public IP address of the EC2 instance."
}

output "instance_id" {
  value       = aws_instance.example.id
  description = "The ID of the EC2 instance."
}

Terraform Module: ec2-module/variables.tf

variable "ami_id" {
  description = "The AMI ID for the EC2 instance."
  type        = string
  default     = "ami-053b0d53c279acc90" # Example: Ubuntu Server 20.04 LTS (HVM), SSD Volume Type
}

variable "instance_type" {
  description = "The instance type for the EC2 instance."
  type        = string
  default     = "t2.micro"
}

variable "key_pair_name" {
  description = "The name of the EC2 Key Pair to use."
  type        = string
  # IMPORTANT: You must have an existing key pair in your AWS account.
  # For testing, you might create a temporary one or use an existing one that supports SSH.
}

variable "vpc_id" {
  description = "The ID of the VPC to deploy into."
  type        = string
  # Default to an example VPC ID or fetch dynamically in test.
  # For simplicity, we'll assume a default VPC is available or provided.
  default     = "vpc-0a1b2c3d4e5f6a7b8" # Replace with a valid VPC ID for your region
}

variable "subnet_id" {
  description = "The ID of the subnet to deploy into."
  type        = string
  # Default to an example Subnet ID or fetch dynamically in test.
  default     = "subnet-0abcdef1234567890" # Replace with a valid Subnet ID for your VPC
}

variable "name_prefix" {
  description = "A prefix for resource names."
  type        = string
  default     = "test-ec2"
}

variable "environment" {
  description = "The environment tag for resources."
  type        = string
  default     = "test"
}

Gemfile (ec2-module/Gemfile)

source "https://rubygems.org"

gem "test-kitchen"
gem "kitchen-terraform"
gem "kitchen-inspec"
gem "train-ssh" # For SSH connectivity to instances

Run bundle install in the ec2-module directory to install dependencies.

Kitchen Configuration (ec2-module/.kitchen.yml)

---
driver:
  name: terraform # Use the kitchen-terraform driver
  # Define variables for your Terraform module here.
  # These will be passed as -var arguments to terraform plan/apply.
  # Ensure `key_pair_name`, `vpc_id`, `subnet_id` are valid for your AWS account/region.
  variables:
    key_pair_name: "your-ssh-key-name" # <-- IMPORTANT: Replace with your actual SSH key pair name
    vpc_id: "vpc-0a1b2c3d4e5f6a7b8"   # <-- IMPORTANT: Replace with a valid VPC ID
    subnet_id: "subnet-0abcdef1234567890" # <-- IMPORTANT: Replace with a valid Subnet ID
    environment: "kitchen-test"
  # You can also pass environment variables if needed
  # env_variables:
  #   AWS_DEFAULT_REGION: "us-east-1"

provisioner:
  name: terraform

verifier:
  name: inspec

platforms:
  - name: aws # This is more of a logical platform name for Kitchen; the driver handles AWS interaction.

suites:
  - name: default
    driver:
      # These parameters are specific to the kitchen-terraform driver
      # They tell the driver where your Terraform root module is.
      root_module_directory: "." # The root of your Terraform module (where main.tf is)
      # You can specify a different Terraform workspace if needed
      # workspace_name: "test"
    verifier:
      # InSpec will connect via SSH to the instance.
      # The IP comes from Terraform outputs.
      # You need the private key associated with 'your-ssh-key-name'.
      inspec_tests:
        - test/integration/default
      ssh_key: "~/.ssh/your-ssh-key.pem" # <-- IMPORTANT: Replace with path to your private key
      host_key_checking: false # Disable for testing, but be cautious in production
      username: ubuntu # Default user for Ubuntu AMIs
      # This tells InSpec where to find the IP address to connect to.
      # It maps to the 'instance_public_ip' output in your main.tf.
      target_uri: "ssh://<%= instance.outputs.instance_public_ip %>"

InSpec Test (ec2-module/test/integration/default/default_spec.rb)

# InSpec test for an EC2 instance
# Learn more about InSpec at: https://www.inspec.io/docs/reference/resources/

# Check if the EC2 instance itself is running (Kitchen-Terraform ensures this by deploying)
# This test focuses on what's configured *inside* the instance.

control 'ec2-instance-configuration' do
  impact 1.0
  title 'EC2 Instance Configuration Check'
  desc 'Verify that the EC2 instance is configured as expected.'

  # Test that a file created by user_data exists
  describe file('/tmp/hello.txt') do
    it { should exist }
    its('content') { should include 'Hello from Terraform' }
  end

  # Test if Nginx service is installed and running
  describe package('nginx') do
    it { should be_installed }
  end

  describe service('nginx') do
    it { should be_installed }
    it { should be_enabled }
    it { should be_running }
  end

  # Test if port 80 (Nginx default) is listening
  describe port(80) do
    it { should be_listening }
    its('protocols') { should include 'tcp' }
  end

  # Test for SSH port (22)
  describe port(22) do
    it { should be_listening }
    its('protocols') { should include 'tcp' }
  end

  # You can add more checks here, e.g.,
  # - specific users/groups
  # - directory permissions
  # - contents of configuration files
  # - environment variables
end

To run these tests:

1. Make sure you’ve installed the Ruby gems (bundle install).
2. From the ec2-module directory, run: bundle exec kitchen test

<