RETROSPECTIVE

June 18th, 2021

SaintsXCTF Version 2.0: AWS Infrastructure

AWS

Terraform

HCL

AWS API Gateway

AWS Lambda

AWS S3

AWS CloudFront

AWS RDS

AWS EC2

s

This is part of a series of articles on SaintsXCTF Version 2.0. The first article in the series provides an overview of the application.

The infrastructure for my website saintsxctf.com is hosted on AWS. SaintsXCTF had AWS infrastructure prior to version 2.0, but it was redesigned and rewritten for the new version. This article outlines the infrastructure and walks through Terraform code which configures and builds the infrastructure.

  • Architectural Overview
  • AWS Infrastructure
  • Kubernetes Infrastructure
  • React Web Application Overview
  • Web Application Redux State Configuration
  • Web Application Cypress E2E Tests
  • Web Application JSS Modular Design
  • Web Application Docker & Nginx Configuration
  • Flask Python API
  • Flask API Testing
  • Flask API Docker & Docker Compose Configuration
  • Function API Using API Gateway & Lambda
  • Auth API Using API Gateway & Lambda
  • Database Deployments Using Jenkins
  • Database Client on Kubernetes
  • IOS Application Updates and Learning Experiences
  • Testing and Continuous Deployment on Jenkins
  • Post-Deployment Challenges & Future Improvements

SaintsXCTF application infrastructure can be grouped into two categories - AWS and Kubernetes. This article only discusses the AWS infrastructure, which has a green background in the diagram below. The Kubernetes infrastructure, which has a red background in the diagram, is discussed in the next article.

In Terraform code, the SaintsXCTF AWS infrastructure is logically grouped into modules. These modules include (but are not limited to) a load balancer to the Kubernetes infrastructure, an application asset S3 bucket, a user asset S3 bucket, an authentication REST API, an AWS Lambda function REST API, and a MySQL RDS database. The other Terraform modules in the SaintsXCTF infrastructure repository perform smaller tasks such as debugging, database backups, and configuring SSL/TLS certificates. These are not discussed in this article.

While designing the SaintsXCTF version 2.0 infrastructure, I put a major emphasis on creating separate production and development environments. Since the initial website version was live during development of version 2.0, I needed a separate non-production configuration to use for testing. Besides for the domain names and AWS resource names, the infrastructure between the production and development environments is identical. This allows for predictable behavior when building the production environment once the development environment is fully tested. I plan to continue using the development environment for testing new infrastructure changes going forward.

Under normal circumstances, the development environment isn't running in order to save money. However, recreating it only requires some terraform apply commands invoked from Jenkins jobs. When the development environment is running, the web application, main API, authentication API, and function API are accessible from dev.saintsxctf.com, dev.api.saintsxctf.com, dev.auth.saintsxctf.com, and dev.fn.saintsxctf.com. Their respective production equivalents are accessible from saintsxctf.com, api.saintsxctf.com, auth.saintsxctf.com, and fn.saintsxctf.com.

As previously mentioned, the SaintsXCTF AWS Terraform infrastructure is logically grouped into modules. All these modules are directories at the top level of the saints-xctf-infrastructure repository. Most modules are configured differently for development and production environments. For example, the authentication API module has the following directory structure.

There are three environmental configurations in this module - one for resources shared across all environments (all), one for development (dev), and one for production (prod). These environmental configurations invoke the submodules, which in the case of the authentication API are api-gateway, lambda, and secrets-manager. For example, inside saints-xctf-com-auth/env/prod is a main.tf file which provisions the production infrastructure through the submodules.

# main.tf provider "aws" { region = "us-east-1" } terraform { required_version = ">= 0.15.0" required_providers { aws = { source = "hashicorp/aws" version = ">= 3.42.0" } } backend "s3" { bucket = "andrew-jarombek-terraform-state" encrypt = true key = "saints-xctf-infrastructure/saints-xctf-com-auth/env/prod" region = "us-east-1" } } module "lambda" { source = "../../modules/lambda" prod = true } module "api-gateway" { source = "../../modules/api-gateway" prod = true authenticate-lambda-name = module.lambda.authenticate-function-name authenticate-lambda-invoke-arn = module.lambda.authenticate-function-invoke-arn token-lambda-name = module.lambda.token-function-name token-lambda-invoke-arn = module.lambda.token-function-invoke-arn } module "secrets-manager" { source = "../../modules/secrets-manager" prod = true rotation-lambda-arn = module.lambda.rotate-function-arn secret_rotation_depends_on = [ module.lambda.rotate-function-arn, module.lambda.rotate-secrets-manager-permission-id, module.lambda.rotate-log-group-id ] }

The Terraform submodules are invoked with certain input variables, including a prod flag. This variable determines whether the infrastructure needs to be built in the production environment (true) or the development environment (false). Most of the modules I talk about in the upcoming segments follow this environment/submodule pattern.

The Terraform module which creates an AWS Application Load Balancer to route traffic to AWS EKS is unique compared to the other modules discussed in this article. Instead of using the AWS Terraform provider to build the infrastructure, the load balancer is created by the Kubernetes Terraform provider. Specifically, the EKS cluster has an ALB Ingress Controller (now known as an AWS Load Balancer Controller) running in a Kubernetes pod. This ingress controller is capable of building load balancers that route traffic to Kubernetes services. Using the Kubernetes Terraform provider, a Kubernetes Ingress object is created with certain annotations, resulting in the ingress controller provisioning load balancer infrastructure in AWS.

In the context of SaintsXCTF, both the web application and main API are hosted on Kubernetes. The load balancer created by the ingress controller routes traffic to Kubernetes for both the saintsxctf.com and api.saintsxctf.com domains. The Terraform module for the load balancer and Kubernetes Ingress object exists in the saints-xctf-com-ingress directory in the saints-xctf-infrastructure repository. I will provide more details about this code in my upcoming article on SaintsXCTF Kubernetes infrastructure.

SaintsXCTF contains images and videos which are hosted in an S3 bucket behind a CloudFront distribution. I refer to this bucket as the "application asset S3 bucket", and it is accessible from asset.saintsxctf.com. I wrote an article about hosting static websites on S3 and CloudFront, and the concept is similar for asset.saintsxctf.com. S3 holds the images and videos as objects, and the CloudFront CDN speeds up the retrieval of these assets.

The Terraform module for the application asset S3 bucket exists in the saints-xctf-com-asset directory in the saints-xctf-infrastructure repository. In this module there is just one production environment configuration. There is no separate development configuration since the assets are the same for both development and production environments. Also, during development, there is no harm in adding additional assets that aren't yet accessible on the production website.

At a high level, the asset.saintsxctf.com module contains a main.tf file with the following resources.

#----------------------- # Existing AWS Resources #----------------------- data "aws_acm_certificate" "wildcard-saintsxctf-com-cert" { domain = "*.saintsxctf.com" } data "aws_acm_certificate" "wildcard-asset-saintsxctf-com-cert" { domain = "*.asset.saintsxctf.com" } data "aws_route53_zone" "saintsxctf" { name = "saintsxctf.com." } #-------------------------------------- # New AWS Resources for S3 & CloudFront #-------------------------------------- resource "aws_s3_bucket" "asset-saintsxctf" {...} resource "aws_cloudfront_distribution" "asset-saintsxctf-distribution" {...} resource "aws_cloudfront_origin_access_identity" "origin-access-identity" {...} resource "aws_cloudfront_distribution" "www-asset-saintsxctf-distribution" {...} resource "aws_route53_record" "asset-saintsxctf-a" {...} resource "aws_route53_record" "www-asset-saintsxctf-a" {...} #------------------- # S3 Bucket Contents #------------------- resource "aws_s3_bucket_object" "saintsxctf-png" {...}

First, existing ACM certificates are retrieved so that HTTPS requests to asset.saintsxctf.com are secured by SSL/TLS certificates. These certificates are used in the aws_cloudfront_distribution resources for building CloudFront distributions. Next, the existing saintsxctf.com Route53 zone is retrieved for use in creating DNS A records for asset.saintsxctf.com and www.asset.saintsxctf.com.

Next comes the new AWS resources. The S3 bucket containing SaintsXCTF assets is created with the aws_s3_bucket resource. There are two CloudFront distributions created for the S3 bucket - one for asset.saintsxctf.com and another for www.asset.saintsxctf.com. These are created with the aws_cloudfront_distribution resource. Each CloudFront distribution also gets a Route53 A record, created by the aws_route53_record resources. The final resources are the S3 bucket objects, which are images and videos. These are created with aws_s3_bucket_object resources.

Similarly to how the asset.saintsxctf.com S3 bucket contains application assets for the SaintsXCTF website, the uasset.saintsxctf.com S3 bucket contains user assets. These assets include profile pictures and group pictures. The Terraform module for the user asset S3 bucket is nearly identical to the application asset S3 bucket. One main difference is user assets (S3 objects) are not declared as resources in the Terraform code. Instead, they are uploaded by users through the website. I will discuss this process in a future article when going over the SaintsXCTF Lambda Function API.

The Terraform module for the application user asset S3 bucket exists in the saints-xctf-com-uasset directory in the saints-xctf-infrastructure repository.

The SaintsXCTF application has four AWS Lambda functions related to authentication, two of which are exposed through a REST API by API Gateway. The premise of the authentication API is to use JWTs to authenticate users of the application. With a proper JWT, users can make API calls and operate within the website and mobile applications.

The Terraform module for the authentication API exists in the saints-xctf-com-auth directory in the saints-xctf-infrastructure repository. The authentication API also has a separate saints-xctf-auth repository for the AWS Lambda function source code.

The infrastructure diagram above shows the four authentication-based AWS Lambda functions. They are SaintsXCTFAuthenticate, SaintsXCTFToken, SaintsXCTFAuthorizer, and SaintsXCTFRotate. SaintsXCTFAuthenticate and SaintsXCTFToken are exposed through the authentication API, while SaintsXCTFAuthorizer and SaintsXCTFRotate are only for internal use.

All four AWS Lambda functions are written in Python. Their general use is as follows:

  1. Upon sign in, the https://auth.saintsxctf.com/token endpoint is invoked, which calls the SaintsXCTFToken function. The user's credentials are passed with this request and the response contains a JWT if the credentials are valid.
  2. Every time an API call that requires authentication is made to https://api.saintsxctf.com, the https://auth.saintsxctf.com/authenticate endpoint is invoked by the API. This endpoint calls the SaintsXCTFAuthenticate function. The JWT token received from https://auth.saintsxctf.com/token is passed in the request body of https://auth.saintsxctf.com/authenticate, and if the JWT is valid the endpoint returns true. Otherwise, it returns false. The SaintsXCTF API handles the response from https://auth.saintsxctf.com/authenticate by proceeding with the user's request if the response is true. Otherwise, it returns an HTTP 403 Forbidden error code.
  3. When requests are made to the https://fn.saintsxctf.com API Gateway REST API which require authorization, they use the SaintsXCTFAuthorizer as a lambda authorizer. The JWT token received from https://auth.saintsxctf.com/token is passed in the Authorization request header to https://fn.saintsxctf.com. If the authorizer function determines that the JWT is valid, it gives the API Gateway endpoint permission to call its corresponding AWS Lambda function. If the JWT is invalid, invoking the AWS Lambda function is forbidden.
  4. The JWT tokens created by the SaintsXCTFToken AWS Lambda function are signed using an RS256 (RSA Signature with SHA-256) algorithm1. To perform this signing, a public key and private key are stored in AWS SecretsManager. For additional security, these public and private keys are rotated on a regular basis with the SaintsXCTFRotate AWS Lambda function. This limits the damage if an unwanted user gains access to the public and private keys, since their ability to sign tokens only lasts until the next rotation.

In a future article I will discuss all the authentication functions in more detail and step through their code.

The SaintsXCTF application has another REST API created with API Gateway and AWS Lambda called the "Function REST API". This API is accessible from https://fn.saintsxctf.com. It includes endpoints for sending emails and uploading pictures. The AWS Lambda functions used by the REST API endpoints are written in Python and JavaScript.

The Terraform module for the function API exists in the saints-xctf-com-fn directory in the saints-xctf-infrastructure repository. The function API also has a separate saints-xctf-functions repository for the AWS Lambda function source code.

As with the authentication API, a future article will be dedicated to the function API.

One main piece of infrastructure that didn't change in SaintsXCTF version 2.0 is the RDS MySQL database. This database holds all the persistent data used by the application, such as user information and exercise logs.

The Terraform module for the RDS MySQL database exists in the database directory in the saints-xctf-infrastructure repository. I also have a private repository which holds SQL scripts for database deployments.

At a high level, the database module contains a main.tf file with the following resources.

resource "aws_security_group" "saints-xctf-database-security" {...} resource "aws_db_instance" "saints-xctf-mysql-database" {...} resource "aws_db_subnet_group" "saints-xctf-mysql-database-subnet" {...} resource "aws_cloudwatch_metric_alarm" "saints-xctf-mysql-database-storage-low-alarm" {...}

aws_db_instance is the main AWS resource that creates the RDS MySQL database. aws_security_group provides other subnets in the SaintsXCTF VPC access to make connections to the database, while aws_db_subnet_group specifies two subnets that the database exists within. Since both subnets are housed in different AWS AZs (Availability Zones), this configuration makes the database highly available. High availability provides protection in case one Availability Zone goes down. Finally, the aws_cloudwatch_metric_alarm resource creates a CloudWatch alarm in case the RDS instance runs low on storage.

In many ways, an application is only as good as the foundation it is built upon. The AWS infrastructure for SaintsXCTF keeps the application running with little downtime, which is key for a good user experience. It is also modularized, allowing me to easily add new modules and drop old ones. This enables future application updates to be smaller in scope than version 2.0, which completely rewrote the entire codebase.

In my next SaintsXCTF article I'll discuss the other crucial piece of infrastructure for the application: Kubernetes on EKS. All the AWS infrastructure code for SaintsXCTF can be found on GitHub.