I started looking into Synthetic Monitoring as a way to test my SaintsXCTF application running in production. I had an issue where the website unexpectedly stopped working, and there was no automated process in place to alert me. With Synthetic Monitoring, I created canary functions to test critical paths of the website, such as signing in a user. If canary functions fail, I get an email alerting me of the issue.
Synthetic Monitoring is a relatively new AWS service, so documentation is a bit lighter when compared to other services. Specifically, documentation for building canary functions with Terraform is incomplete with very few code samples. In this article, I give an overview of my Synthetic Monitoring AWS infrastructure and show how it's configured with Terraform. I also give a brief walk through of my canary function source code. All the code mentioned in this article is available on GitHub.
My Synthetic Monitoring infrastructure consists of canary functions, an S3 bucket holding canary function results and images, and CloudWatch events to notify me via email when a canary function fails. This infrastructure is shown in the diagram below.
The following Terraform code creates one of my canary functions. This canary function tests the sign in functionality on my website. It also sets up a CloudWatch event for sending alerts when the function fails.
The canary function Terraform module code is available in a main.tf file on GitHub.
The results of the canary function are uploaded to an S3 bucket, which is specified in the artifact_s3_location argument on the aws_synthetics_canary resource. This S3 bucket is created with the following configuration:
The S3 bucket Terraform module code is available in a main.tf file on GitHub.
Canary functions are given an IAM role, which is specified in the execution_role_arn argument on the aws_synthetics_canary resource. The following code creates this IAM role and its corresponding policy.
There are certain permissions that canary functions must have in their IAM roles1. These permissions are specified in the CanaryGeneric statement of the policy. My canary functions also require additional permissions, specifically for AWS Secrets Manager. These permissions are specified in the CanarySecretsManager statement of the policy.
The IAM Terraform module code is available in a main.tf file on GitHub.
After all this infrastructure is created, the canary functions and their execution results are viewable in the AWS console. Individual functions can also be viewed and their execution results can be thoroughly analyzed.
This function uses the synthetics library, which is imported with const synthetics = require('Synthetics'). The synthetics library is a wrapper around Puppeteer, so reading the AWS Synthetics documentation along with the Puppeteer documentation is enough to hit the ground running. Synthetics also has a bunch of custom functions, such as the executeStep() function used in my code. executeStep() runs Puppeteer commands and takes screenshots of the website, which are eventually uploaded to the canary function's S3 bucket.
The canary function starts by navigating to the sign in page. This is performed by the page.goto('https://saintsxctf.com/signin') command. Then, in the four executeStep() functions, it types in credentials to a sign in form ('enter_credentials'), clicks the sign in button ('sign_in'), clicks on the profile page button once signed in ('profile_page_click'), and checks to see if an element appears in the profile page ('profile_page_view').
This code also shows how canary functions can use the AWS SDK, in this case to grab user credentials from Secrets Manager. Secrets Manager helps avoid the bad practice of hard coding credentials into the canary function source code.
Another runtime option for canary functions is Python and the Selenium library. My Python canary function tests whether a "forgot password" email can be sent from my website. The source code is shown below, and exists in a forgot_password.py file.
Synthetic Monitoring canary functions are a nice addition to AWS that allow users to easily create and schedule browser test code for applications. In my case, canary functions assist in sending me notifications if critical parts of my website stop working. Although documentation is still a bit rough around the edges, I believe canary functions are a worthwhile option for end to end testing applications running in production environments. All the source code for this article is available on GitHub.