From October 2018 to May 2019, I moved the infrastructure for both my websites to AWS. The process for building and tearing down this infrastructure is automated with IaC, specifically Terraform. I've had a lot of fun working with Terraform and learning the different design patterns for infrastructure in the cloud.
After my infrastructure was built, I realized I needed a way to test that my IaC was behaving as expected. The obvious solution to this requirement was a unit test suite. I implemented this unit test suite in Python with the help of the AWS SDK. This article explains why I took the time to write unit tests and walks through of the basics of testing AWS infrastructure in Python.
Over the three and a half years I've been a software engineer, I've slowly realized how valuable unit tests are. In my early applications, unit tests were noticeably absent. Unit tests finally started appearing in my repositories this spring. Now it is mandatory for new applications to have full unit test coverage and I've begun adding tests to older applications.
A Unit Test is an assertion that a unit of code is working as expected1. Units of code can be a single line, a function, a class, or an application. Unit tests are run on a regular basis. This includes (but is not limited to) code commits, application deployments, and scheduled intervals.
There are three main reasons why I hopped on the unit test bandwagon. The first reason is that writing unit tests helps catch poorly written or useless code. When I write code, my brain tends to wander and get sidetracked. Often I write a section of code over the course of days or weeks. I usually don't have the time to go back through every line of code and make sure its well written or still being used. However, when I write unit tests and have to formulate test scenarios for every line of a program, smelly code becomes easy to identify.
The second reason is that unit tests help catch recurring bugs before they make it into a production build. A good practice is to create a unit test every time a bug is found in an application. Writing tests for a bug helps developers learn their root cause and provides quick detection for their return in the future.
The third reason is that unit tests ease the pain of upgrading technology versions in an application, whether it be a language, framework, or library. Unit tests give us peace of mind that all the corners of our application are still functional after software upgrades occur.
I firmly believe the upfront costs of building unit tests are worthwhile in the long run.
I wrote unit tests for my Terraform infrastructure using Python and the boto3 AWS SDK. I took two different approaches for setting up these unit tests. For my global and SaintsXCTF AWS infrastructure, I reinvented the wheel and created custom test suites. By the time I wrote my jarombek.com infrastructure, I figured there must be a better approach. I decided to scrap the custom test suite and used Python's built-in unittest library instead. I'll demonstrate both approaches next, followed by some unit test function examples.
My first approach to unit testing infrastructure involved reinventing the wheel in regards to building a test suite. I first defined an entrypoint to the unit tests. Executing a single Python file runs all the unit tests I wrote:
This entrypoint file is referred to as the "master test suite." The master test suite invokes child test suites which are grouped by AWS resource type. For example, all my ACM (AWS Certificate Manager) infrastructure tests are found in
acmTestSuite and are invoked by calling
Notice that the master test suite and ACM test suite have the same structure. Both test suites contain a list of tests and a function which executes the tests. The main difference is that the master test suite executes a list of test suites while the ACM test suite executes a list of unit tests. Test suites are executed with the
Test.testsuite() function and unit tests are executed with the
A nice aspect of my custom approach is that a child test suite can be run as if it's the master test suite. In this scenario, only the tests within the child test suite will execute.
However, reinventing the unit test wheel didn't add much value to my tests. Therefore, I decided to try a different approach for my jarombek.com infrastructure tests.
My second approach to unit testing infrastructure involved the built-in Python unittest framework. unittest has an additional component called a test runner. The test runner orchestrates test suites and the execution of unit tests inside them2. My test runner is configured to execute all the test suites.
In unittest, test suites are subclasses of
TestCase provides assertion methods, setup and teardown methods, and test execution methods3. The following class is a test suite for ACM infrastructure tests:
With unittest, much of the test suite setup code necessary in my custom approach is handled by the test framework. This allows me to spend more time working on the unit tests themselves.
If you want to explore the unit tests I created in depth, I recommend exploring my GitHub repositories. As a quick example, the following two code snippets are my ACM tests for saintsxctf.com and jarombek.com. The first is written using my custom unit test approach and the second is written using the unittest framework.
If I could start my infrastructure unit tests over from scratch, I would write them all with the unittest framework. I believe that developers should only reinvent the wheel under two scenarios - as a learning experience or if they think they can improve the existing technology. While it was a good learning experience creating a unit test suite from scratch, it was very barebones functionality wise. Because of this, it didn't assist me in learning the underworkings of test frameworks such as unittest.
I should've spent more time exploring the different approaches to Python unit tests before jumping into a custom solution. There a many different unit testing frameworks for each programming language, and I will make sure to explore all my options before choosing one in future projects.