Infrastructure Testing: Terratest & Kitchen for Terraform Modules

As a senior engineer at Khadervali.com, I’ve seen firsthand how crucial robust testing is in every facet of software development. Yet, for many teams, infrastructure as Code (IaC) remains a significant blind spot. We meticulously test application code, run unit, integration, and end-to-end tests, but when it comes to the underlying infrastructure, the process often reverts to manual verification or, at best, rudimentary checks.

This is a precarious position. Your applications are only as reliable as the infrastructure they run on. A misconfigured security group, an improperly provisioned database, or an incorrect IAM policy can lead to outages, security breaches, and significant operational headaches. Manually deploying and verifying Terraform modules, even in staging environments, is slow, error-prone, and doesn’t scale. It’s time to bring the same rigor of application testing to our infrastructure.

In this comprehensive guide, we’ll dive deep into two powerful tools that enable sophisticated, automated infrastructure testing for Terraform modules: Terratest and Kitchen-Terraform. We’ll explore their philosophies, capabilities, and how to effectively integrate them into your development workflow, even showing how to combine their strengths for ultimate reliability.

The Paradigm Shift: From Manual Verification to Automated Validation

Think about the typical lifecycle of a Terraform module development without automated testing:

1. Write Terraform configuration.
2. Run terraform plan to review changes (often just a quick glance).
3. Run terraform apply in a dev/staging environment.
4. Manually log into the cloud console (AWS, Azure, GCP) or use the CLI to verify resources.
5. Perhaps run some basic connectivity tests or application smoke tests.
6. If issues arise, debug, fix, and repeat the manual process.
7. Eventually, promote to production, hoping no subtle misconfigurations slipped through.

This “deploy and pray” approach is fundamentally flawed. It’s time-consuming, prone to human error, and lacks reproducibility. Each manual verification step is a potential point of failure. The further down the pipeline an error is found, the more expensive it is to fix.

The solution lies in shifting left – integrating testing early and continuously into the IaC development process. Automated infrastructure testing brings several transformative benefits:

• Increased Confidence: Know that your infrastructure behaves exactly as intended before it hits production.
• Faster Feedback Loops: Catch errors within minutes, not hours or days.
• Reduced Human Error: Automate repetitive verification tasks.
• Improved Collaboration: Tests serve as executable documentation for module behavior.
• Enhanced Reliability & Security: Prevent misconfigurations that lead to outages or vulnerabilities.
• Simplified Refactoring: Make changes to modules with confidence, knowing tests will catch regressions.

Terratest and Kitchen-Terraform are two distinct but complementary approaches to achieving this automated validation, each with its own strengths and ideal use cases. Let’s explore them.

<

>

Understanding Terratest: Go-based End-to-End Testing for Infrastructure

What is Terratest?

Terratest is an open-source Go library that provides a framework for writing automated tests for your infrastructure code. Developed by Gruntwork, it allows you to:

• Deploy real infrastructure (e.g., AWS, Azure, GCP, Kubernetes) using Terraform, CloudFormation, Packer, Helm, etc.
• Use the native cloud provider SDKs (e.g., AWS SDK for Go) to query the deployed infrastructure and validate its configuration and behavior.
• Run commands on remote servers (e.g., SSH, execute scripts).
• Make HTTP requests to test web services.
• Tear down the infrastructure after tests complete, ensuring a clean state.

Because Terratest is a Go library, you write your tests as standard Go programs, leveraging Go’s robust testing ecosystem (`go test`), rich standard library, and powerful third-party assertion libraries (like testify/require or testify/assert).

Why Go for Infrastructure Testing?

• Concurrency: Go’s goroutines make it easy to run tests in parallel, speeding up execution.
• Performance: Compiled Go binaries are fast, crucial for tests that interact with cloud APIs.
• Strong Typing: Reduces runtime errors and improves code maintainability.
• Rich Ecosystem: Excellent integration with cloud provider SDKs (e.g., aws-sdk-go), HTTP clients, and other utilities.
• Developer Familiarity: Go is increasingly popular in the DevOps and cloud-native space.

Basic Terratest Structure

A typical Terratest workflow involves these steps, often orchestrated within a single Go test function:

1. Setup: Define Terraform options (paths, variables).
2. Deployment: Use terraform.InitAndApply to deploy your module.
3. Validation:
   • Retrieve outputs from Terraform using terraform.Output.
   • Use cloud SDKs (e.g., aws.GetS3BucketTags) to query the deployed resources.
   • Assert expected values and behaviors using Go’s testing package and assertion libraries.
4. Teardown: Use defer terraform.Destroy to ensure resources are cleaned up, regardless of test outcome.

The test_structure package is particularly useful for managing test state across multiple test functions or stages, allowing you to persist outputs or other data between setup, deployment, and validation phases.

Code Example 1: Simple S3 Bucket Test with Terratest

Let’s create a simple Terraform module for an S3 bucket and then write a Terratest suite to validate its creation, tags, and public access block settings.

Terraform Module (`modules/s3-bucket/main.tf`):

# modules/s3-bucket/main.tf
resource "aws_s3_bucket" "this" {
  bucket = var.bucket_name
  tags   = var.tags
}

resource "aws_s3_bucket_public_access_block" "this" {
  bucket = aws_s3_bucket.this.id

  block_public_acls       = var.block_public_acls
  block_public_policy     = var.block_public_policy
  ignore_public_acls      = var.ignore_public_acls
  restrict_public_buckets = var.restrict_public_buckets
}

variable "bucket_name" {
  description = "Name of the S3 bucket"
  type        = string
}

variable "tags" {
  description = "Tags for the S3 bucket"
  type        = map(string)
  default     = {}
}

variable "block_public_acls" {
  description = "Whether to block public ACLs"
  type        = bool
  default     = true
}

variable "block_public_policy" {
  description = "Whether to block public policies"
  type        = bool
  default     = true
}

variable "ignore_public_acls" {
  description = "Whether to ignore public ACLs"
  type        = bool
  default     = true
}

variable "restrict_public_buckets" {
  description = "Whether to restrict public buckets"
  type        = bool
  default     = true
}

output "bucket_id" {
  description = "The name of the S3 bucket"
  value       = aws_s3_bucket.this.id
}

Terratest Go Test (`test/s3_bucket_test.go`):

// test/s3_bucket_test.go
package test

import (
	"fmt"
	"testing"
	"time"

	"github.com/gruntwork-io/terratest/modules/aws"
	"github.com/gruntwork-io/terratest/modules/random"
	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/stretchr/testify/require"
)

func TestS3BucketModule(t *testing.T) {
	t.Parallel() // Allows tests to run in parallel

	// Generate a unique name for the S3 bucket to ensure isolation
	expectedBucketName := fmt.Sprintf("khadervali-test-bucket-%s", random.UniqueId())
	expectedTagKey := "Environment"
	expectedTagValue := "Test"

	// Specify the path to the Terraform module being tested.
	terraformDir := "../modules/s3-bucket"

	// Define Terraform options.
	terraformOptions := &terraform.Options{
		TerraformDir: terraformDir,
		Vars: map[string]interface{}{
			"bucket_name": expectedBucketName,
			"tags": map[string]string{
				expectedTagKey: expectedTagValue,
				"Project":      "KhadervaliBlog",
			},
			"block_public_acls":       true,
			"block_public_policy":     true,
			"ignore_public_acls":      true,
			"restrict_public_buckets": true,
		},
		// The region to deploy the infrastructure to.
		// Make sure your AWS credentials are configured for this region.
		AwsRegion: aws.Get --> (t),
	}

	// At the end of the test, run `terraform destroy` to clean up any resources that were created.
	defer terraform.Destroy(t, terraformOptions)

	// Run `terraform init` and `terraform apply`. Fail the test if there are any errors.
	terraform.InitAndApply(t, terraformOptions)

	// Get the bucket ID output from Terraform.
	bucketID := terraform.Output(t, terraformOptions, "bucket_id")

	// Verify the S3 bucket exists and has the correct tags.
	aws.AssertS3BucketExists(t, aws.GetDefaultRegion(t), bucketID)

	// Get the tags for the S3 bucket.
	tags := aws.GetS3BucketTags(t, aws.GetDefaultRegion(t), bucketID)

	// Assert that the expected tag exists with the correct value.
	require.Contains(t, tags, expectedTagKey)
	require.Equal(t, expectedTagValue, tags[expectedTagKey])

	// Verify public access block settings.
	// We need to retrieve the actual public access block configuration.
	// The AWS SDK for Go provides functions for this.
	// For simplicity, we'll use a direct check for the default region for this example.
	// In a real scenario, you might use specific AWS SDK calls.
	// Note: aws.GetS3BucketPublicAccessBlock is not directly available in terratest/modules/aws.
	// You would typically use the underlying aws-sdk-go client directly for granular checks.
	// For demonstration, let's assume `aws.GetS3BucketPublicAccessBlock` exists or we mock it.
	// Or, more practically, we can assert based on the overall security implications.
	// For this example, let's assume the module's default values for public access blocks are correct.
	// A more robust check would involve using the actual AWS SDK.

	// Example using aws-sdk-go directly for a more specific check (requires import "github.com/aws/aws-sdk-go/service/s3")
	// s3Client := aws.NewS3Client(t, aws.GetDefaultRegion(t))
	// input := &s3.GetPublicAccessBlockInput{Bucket: &bucketID}
	// output, err := s3Client.GetPublicAccessBlock(input)
	// require.NoError(t, err)
	// require.True(t, *output.PublicAccessBlockConfiguration.BlockPublicAcls)
	// require.True(t, *output.PublicAccessBlockConfiguration.BlockPublicPolicy)
	// require.True(t, *output.PublicAccessBlockConfiguration.IgnorePublicAcls)
	// require.True(t, *output.PublicAccessBlockConfiguration.RestrictPublicBuckets)

	// For Terratest's level of abstraction, asserting existence and tags is a common starting point.
	// Further checks would use the AWS SDK directly as shown commented above or through custom Terratest helpers.

	t.Logf("S3 bucket '%s' created and validated successfully.", bucketID)
}

// Helper to get a default AWS region, or a random one if not specified by env var
func GetDefaultRegion(t *testing.T) string {
	region := aws.Get || (t)
	if region == "" {
		allRegions := aws.Get = (t, nil) // Get all regions that support S3
		if len(allRegions) > 0 {
			region = allRegions[random.RandomInt(0, len(allRegions))].Name
		} else {
			region = "us-east-1" // Fallback
		}
	}
	t.Logf("Using AWS region: %s", region)
	return region
}

To run this test:

1. Ensure you have Go installed.
2. Ensure you have Terraform installed.
3. Set up your AWS credentials.
4. Create the directory structure: project-root/modules/s3-bucket/main.tf and project-root/test/s3_bucket_test.go.
5. Navigate to the test directory.
6. Run go mod init test (if not already initialized).
7. Run go get github.com/gruntwork-io/terratest/modules/... and go get github.com/stretchr/testify/require.
8. Execute: go test -v

This test will provision a real S3 bucket, verify its existence and tags, and then tear it down. This is the essence of end-to-end infrastructure testing with Terratest.

Deep Dive into Kitchen-Terraform and Test Kitchen

What is Test Kitchen?

Test Kitchen is an integration testing framework that originated in the Chef community. Its primary purpose is to test infrastructure code (like Chef cookbooks or Puppet modules) on various operating systems and platforms, in an isolated and automated manner. It uses a declarative configuration (`.kitchen.yml`) to define:

• Drivers: How the test environment is created (e.g., EC2, Docker, Vagrant).
• Provisioners: How the code under test is applied to the environment (e.g., Chef-client, Puppet apply).
• Verifiers: How the environment is validated (e.g., InSpec, Serverspec).
• Platforms: The target operating systems (e.g., Ubuntu 20.04, Amazon Linux 2).
• Suites: Specific configurations of the code under test, combined with platforms and verifiers.

How Kitchen-Terraform Extends It for Terraform

Kitchen-Terraform is a plugin for Test Kitchen that allows you to use Terraform as the “provisioner.” Instead of Chef or Puppet, Test Kitchen leverages Terraform to provision your infrastructure. This means you can:

• Define your Terraform module as the code under test.
• Use a Test Kitchen driver (e.g., kitchen-terraform‘s built-in driver, or kitchen-ec2) to manage the lifecycle of the test environment.
• Apply your Terraform module within that environment.
• Use a Test Kitchen verifier (most commonly InSpec) to validate the created infrastructure against a set of compliance and security policies.

Kitchen-Terraform essentially bridges the gap, allowing Test Kitchen’s robust isolation and verification capabilities to be applied directly to Terraform modules.

Key Concepts: `kitchen.yml` Explained

The heart of Test Kitchen is the .kitchen.yml file. Here’s a breakdown of its typical structure for Kitchen-Terraform:

# .kitchen.yml
driver:
  name: terraform # The kitchen-terraform driver
  # Any driver-specific options, e.g., variables, backend configuration
  variable_files:
    - test/fixtures/my_module/terraform.tfvars
  backend_configurations:
    bucket: "my-terraform-state-bucket"
    key: "kitchen-terraform-test.tfstate"
    region: "us-east-1"

provisioner:
  name: terraform # This is actually the "driver" for Terraform (confusing, I know)
  # Any provisioner-specific options
  directory: test/fixtures/my_module # Path to your Terraform module

verifier:
  name: inspec # The verifier to use for assertions
  # Verifier-specific options
  inspec_directory: test/integration # Path to InSpec profiles

platforms:
  # Platforms define the target cloud/OS environment where Terraform will run.
  # For Kitchen-Terraform, these often represent different Terraform input variable sets
  # rather than distinct OS images.
  - name: default
    # You can specify AWS region here if needed for the platform context
    driver_config:
      region: "us-east-1"

suites:
  # Suites define combinations of the provisioner, verifier, and platform,
  # allowing different test scenarios.
  - name: minimal
    # Terraform variables specific to this suite
    vars:
      instance_type: t2.micro
      ami_id: "ami-0abcdef1234567890" # Example AMI
    # InSpec profiles specific to this suite
    verifier:
      inspec_tests:
        - test/integration/minimal-instance
  - name: advanced
    vars:
      instance_type: m5.large
      enable_monitoring: true
    verifier:
      inspec_tests:
        - test/integration/advanced-instance

• driver: Specifies how Test Kitchen manages the test lifecycle. For Kitchen-Terraform, this is typically terraform itself, which orchestrates the calls to the Terraform CLI.
• provisioner: Tells Test Kitchen what to apply to the test environment. Again, for Kitchen-Terraform, this is terraform, pointing to your Terraform module.
• verifier: Defines the tool used to validate the infrastructure after it’s provisioned. InSpec is the go-to for Kitchen-Terraform due to its declarative, human-readable syntax for checking cloud resources and operating system states.
• platforms: Represents the environments your module will be deployed into. For Terraform, this usually translates to different variable sets or regions.
• suites: A suite combines a provisioner, verifier, and platform with specific configurations (e.g., different variable inputs for your Terraform module) to create a distinct test scenario.

Architecture Diagram (in words) for Kitchen-Terraform

Imagine the flow when you run kitchen test:

+-------------------+
|   Developer / CI  |
| (runs 'kitchen')  |
+-------------------+
        |
        v
+-------------------+
|    Test Kitchen   |
|  (.kitchen.yml)   |
+-------------------+
        | (1. Creates temporary workspace)
        v
+-------------------+
|  Kitchen-Terraform |
|      (Driver)     |
+-------------------+
        | (2. Runs 'terraform init', 'terraform apply' for suite)
        v
+-------------------+
|   Terraform CLI   |
+-------------------+
        | (3. Provisions real infrastructure in Cloud/VM)
        v
+-------------------+
|    Cloud Provider |
| (AWS, Azure, GCP) |
|   (EC2, S3, RDS)  |
+-------------------+
        | (4. Gathers Terraform outputs for verifier)
        v
+-------------------+
|  Kitchen-Terraform |
|      (Driver)     |
+-------------------+
        | (5. Invokes InSpec with connection info and outputs)
        v
+-------------------+
|       InSpec      |
|     (Verifier)    |
+-------------------+
        | (6. Connects to Cloud/VM, runs assertions)
        v
+-------------------+
|  Cloud Provider / |
|  OS (Checks APIs, |
|   SSH to VMs)     |
+-------------------+
        | (7. Reports results back to Test Kitchen)
        v
+-------------------+
|    Test Kitchen   |
+-------------------+
        | (8. Runs 'terraform destroy' to clean up resources)
        v
+-------------------+
|   Developer / CI  |
|   (Test Results)  |
+-------------------+

This diagram illustrates how Test Kitchen, through its Terraform driver, orchestrates the entire lifecycle: provisioning the infrastructure with Terraform, then verifying it with InSpec, and finally tearing it down.

Terratest vs. Kitchen-Terraform: Choosing the Right Tool

Both Terratest and Kitchen-Terraform are excellent tools for infrastructure testing, but they excel in different areas. Understanding their strengths helps you choose the right tool for the job.

Terratest Strengths

• Go-Native Flexibility: As a Go library, Terratest gives you the full power of a programming language. This is invaluable for complex scenarios where you need to perform conditional logic, iterate, or interact with multiple cloud services in a sophisticated sequence.
• Direct Cloud SDK Interaction: Terratest allows direct use of AWS, Azure, GCP SDKs. This is perfect for validating not just resource existence but also detailed configurations, permissions, and interactions between services (e.g., “Does this EC2 instance’s IAM role have permission to write to this S3 bucket?”).
• End-to-End Scenarios: Ideal for testing entire application stacks or complex cross-service integrations where you need to provision multiple components and then test their interoperability.
• Robust Helper Functions: Provides a rich set of helper functions for common tasks like running Terraform, checking HTTP endpoints, SSHing into instances, and interacting with Kubernetes.
• Strong for Programmatic Validation: If your validation logic requires more than simple declarative checks, Terratest’s programmatic nature shines.

Kitchen-Terraform Strengths

• Declarative Simplicity: The .kitchen.yml and InSpec profiles are highly declarative. This can make tests easier to read and write for less complex scenarios, especially for teams already familiar with Test Kitchen or Ruby-based testing.
• Excellent Module-Level Testing: It’s particularly strong for testing individual Terraform modules, ensuring they provision resources correctly and adhere to basic security and configuration standards.
• InSpec/Serverspec Integration: If you need to perform deep operating system-level checks (e.g., “Is Nginx installed and running?”, “Are specific ports open?”, “Are certain files present?”), InSpec provides a powerful and readable DSL for this, which integrates seamlessly with Kitchen-Terraform.
• Isolation & Parallelization: Test Kitchen is built for isolation and can run multiple suites in parallel, providing efficient testing of different module configurations.
• Tooling Maturity: Test Kitchen has a mature ecosystem and community, especially around verification tools like InSpec.

When to Use Which (or Both)

• Use Terratest when:
  • You need to test complex interactions between multiple cloud resources or services.
  • Your validation logic requires sophisticated programmatic checks (e.g., polling for a service to become available, simulating user actions, fine-grained permission checks).
  • You’re already comfortable with Go or prefer a programming language for your test automation.