Infrastructure Testing Frameworks

I’ve been taking a look at some infrastructure testing frameworks recently, and also trying to wrap my head around the concept in the first place.

To begin, what is an infrastructure test framework, and why would I need one? As developers, we’re used to writing tests for our code. Once we are ready to deploy that code, it makes sense that we would want to ensure our deployment went as expected, and that our production environment is properly configured and secured. An infrastructure testing framework makes this checking less of a manual process by letting us write runnable tests to verify properties of our server. Some frameworks like Serverspec and terratest are simply code libraries that can be called from your test code. Other frameworks such as goss or InSpec provide a standalone binary making it simpler to run your infra tests, at the cost of not having total flexibility in your test code. Here’s a runthrough of the frameworks I’ve checked out so far:

Serverspec

Started in 2013, Serverspec is the oldest framework on this list, and the project which all the other frameworks on this list point to as an inspiration. To get started with Serverspec you write BDD-style tests using RSpec and then call the Serverspec library to make assertions on your server instance. Serverspec can execute assertions either on the local machine it is running on, or a remote server via ssh.

With Serverspec, you get a lot of flexibility from writing tests in a full-fledged programming language. The built-in set of resource types is not the largest and will probably need to be extended for your needs, but the biggest limitation I found is that each test suite has to be scoped to a single server.

Goss

Goss is a tool inspired by Serverspec, whose tests are written in declarative YAML. Like Serverspec, goss is focused on validating a single server instance, however it does not have support for executing assertions over ssh, so you will need to deploy the goss binary and any test files onto the actual server you wish to validate.

Goss only runs on Linux and probably has the smallest set of built-in tests among all the frameworks listed here, but it comes with a few innovative features:

1. automatic discovery of test cases - the “add” command will scan the system state and use it to create a test case. For example, goss add port 80 will add a test case to check if port 80 is listening if there is already a listener on port 80, otherwise it outputs a test case to check there isn’t a listener.
2. spin up a health check endpoint, which you can then query periodically to ensure that the server remains compliant

Since goss is distributed as a static binary, extending goss with more test types will involve either forking the goss source code or using it as a library for your own CLI tool.

Terratest

Terratest has some similarities with Serverpec in that it ships as a Go library, and you can call it from your regular Go test code. What Terratest offers is an extensive set of integrations with other tools like terraform, packer, AWS, and so on, that you can use as helpers in your test fixtures or test logic to set up actual infrastructure. A typical Terratest test case making use of terraform might go like this:

Terraform file

resource "aws_instance" "web" {
  ami           = "${data.aws_ami.web.id}"
  instance_type = "t2.micro"

  tags {
    Name = "HelloWorld"
  }
}

output "web_address" {
  value = "${aws_instance.web.public_dns}"
}

http_test.go

package test

import (
	http_helper "github.com/gruntwork-io/terratest/modules/http-helper"
	"github.com/gruntwork-io/terratest/modules/terraform"
	"net/http"
	"testing"
)

func TestWeb(t *testing.T) {
	// 1. configure terraform path
	terraformOptions := &terraform.Options{TerraformDir: "../terraform-web"}

	// 2. ensure tear down of test infrastructure
	defer terraform.Destroy(t, terraformOptions)

	// 3. set up test infrastructure
	terraform.InitAndApply(t, terraformOptions)

	// 4. Write test cases
	addr := terraform.Output(t, terraformOptions, "web_address")
	status, _ := http_helper.HttpGet(t, "http://"+addr)
	if status != http.StatusOK {
		t.Errorf("expected %d, got %d", http.StatusOK, status)
	}
}

One caveat of using terratest is that there are some default behaviours in go test that will disrupt long-running tests, but the necessary configuration is is documented in the Best Practices section.

InSpec

InSpec is a test framework specifically geared towards executing compliance controls. Although it is a fork of Serverspec, InSpec has more of a focus on detecting and remediating compliance failures, and the community already maintains several baselines that can serve as a starting point for your own requirements.

One major difference with InSpec is that 4.x binary distributions require a licensing agreement for commercial use. To clarify, the actual source code is still Apache 2.0-licensed, but to use the official binaries distributed by Chef requires a commercial license (for this article, I evaluated version 3.9.3 which doesn’t have the requirement.)

While looking into InSpec I quickly found that it was by far the most polished of all the frameworks here. The InSpec home page is rich with tutorials, case studies, blog posts, and is also supported by the existing Chef community. The documentation reference is also well organised, and out of the box the InSpec tool comes with the most comprehensive set of resource types.

Note: Although it is maintained by the company behind Chef, InSpec does not force you to use Chef automation in any way.

Summary

To keep this blog post short, I’ve created a separate repository with concrete examples of each framework above.

Here’s a table summarizing some of the differences between frameworks:

Ultimately, all of these frameworks still seem to be in popular usage, and their choice depends on your current requirements - InSpec is probably the best choice for compliance testing, but may be overkill if you are just looking for an easy way to verify your deployments, in which case you might want to consider a framework whose language you are most familiar with like Serverspec for ruby shops or Terratest for golang (for python you want to consider Testinfra). Lastly, if your testing needs are simple then goss might be a good choice.