On December 24th 2016, I released my first website saintsxctf.com. I was still a senior in college at the time, and used my limited software development knowledge from classes and a summer internship to build the application. SaintsXCTF is a training log designed for my college Cross Country and Track & Field teams at St. Lawrence University. Running competitively in college had a major impact on my life, and I was really proud to create the website to assist my teammates and coaches. Shortly after releasing the website, I created an Android application and an iOS application for SaintsXCTF. With SaintsXCTF accessible via web browsers and mobile applications, I felt my development work was complete and moved on to other programming projects.
As I began my professional software engineering career in the summer of 2017, I gradually learned industry best practices and became more well rounded as a developer. At this point, certain shortcomings and misguided assumptions about my SaintsXCTF applications became apparent. First, the core web application and API did not follow the latest industry standards. Second, all three applications were not properly tested and were prone to degradation if left unchecked. Third, the security & infrastructure of the applications was very basic and not fault tolerant. Lastly, my assumption that releasing the applications meant my work was done proved to be incorrect. As all software engineers know, the work just begins when an application is initially released.
These realizations resulted in a multi-year effort to create a new version of SaintsXCTF that fixed all the issues with the original. I decided upon a two step process to get the original version of SaintsXCTF converted to the new version. The first step began in December 2018 and was completed in February 2019. This step moved the website, API, and database infrastructure to AWS. All the infrastructure was written as code using Terraform. The second step began in June 2019 and was completed in May 2021. This two year development step rewrote the SaintsXCTF web application, API, and infrastructure. It also created a new authentication API and AWS Lambda Function API.
This article explores the architectural changes to the SaintsXCTF application across these iterations. Subsequent articles discuss certain components, technologies, and design decisions in SaintsXCTF 2.0.
- Architectural Overview
- AWS Infrastructure
- Kubernetes Infrastructure
- React Web Application Overview
- Web Application Redux State Configuration
- Web Application Cypress E2E Tests
- Web Application JSS Modular Design
- Web Application Docker & Nginx Configuration
- Flask Python API
- Flask API Testing
- Flask API Docker & Docker Compose Configuration
- Function API Using API Gateway & Lambda
- Auth API Using API Gateway & Lambda
- Database Deployments Using Jenkins
- Database Client on Kubernetes
- IOS Application Updates and Learning Experiences
- Testing and Continuous Deployment on Jenkins
- Post-Deployment Challenges & Future Improvements
The initial SaintsXCTF infrastructure was extremely barebones, consisting of a single hosted linux server. This server, a virtual machine hosted by Linode, ran the web application, API, and database.
There are many flaws with this architecture design. The most obvious issue is that the linux server is a single point of failure. If the server goes down, so does the website, API, and database. Worst of all, in the case of a catastrophic failure where the virtual machine’s file system isn’t recoverable, all the application data is lost. For an application with user information and exercise logs uploaded every day, this is an unacceptable level of risk.
In the API technology ecosystem, the lack of a server-side framework resulted in hard to read code. Also, certain actions such as creating classes that represent HTTP request objects were written by me. If I used a back-end framework instead, these classes would have been provided for me. I was reinventing the wheel without realizing it. Another issue with the API was that it shared the same codebase and build process (a manual SFTP upload to the linux server) as the front-end code. This meant if the website went down, the API would as well. If new UI code was deployed, the API was redeployed as well. In the second version of the application, I redesigned the codebase to separate the UI and API.
Due to all these flaws in the initial website’s technology stack and infrastructure, I stopped enhancing the functionality of the application shortly after releasing it. Instead, I began planning an iterative process of refurbishing the application up to industry standards.
Before creating entirely new codebases for the UI and API, I fixed all the fatal flaws with the initial version of the application. The biggest flaw of the initial design was the linux server, which was a single point of failure. If that server went down, everything would have been lost.
To remedy this issue I redesigned the infrastructure on AWS and wrote the infrastructure as code using Terraform. As a result, I was able to build the infrastructure in a repeatable manner. This new infrastructure was designed without any single points of failure.
From the end user perspective, nothing changed while using the application after these changes. However, the reliability of the application greatly increased. First, in a worst case scenario where the infrastructure broke, Terraform scripts were capable of recreating the infrastructure in a matter of minutes.
Second, I implemented an AWS Lambda function (not shown in the infrastructure diagram) which takes database backups daily and uploads them to S3. Therefore, the new worst case scenario would result in one day of lost data; a great improvement over losing everything.
Luckily, this scenario is extremely unlikely. AWS RDS (the MySQL database) and AWS EC2 (the API and web application server) have impressive uptime percentages of 99.95% and 99.99%, respectively1,2. RDS also takes snapshots on a regular basis for disaster recovery purposes. With this impressive reliability, I never had to use my database backup files to restore the database. Nevertheless, the peace of mind it gave me was worth it.
With the application infrastructure in a reliable state, I took my time creating a second version of the web application and API. With increased knowledge of cloud infrastructure, containers, and container orchestrators, I also decided to rewrite the application infrastructure for a third time. I released this new codebase at the end of May 2021.
The updated infrastructure for the new version of the application is still hosted on AWS and written as code with Terraform. The biggest infrastructure change was moving to Kubernetes on EKS. Both the API and web application are deployed on Kubernetes pods, with traffic routed to them from an AWS load balancer. All the Kubernetes infrastructure is created with Terraform as well. With the combined ease of Terraform and elegance of Kubernetes deployments, updating or reverting a website or API version is as easy as building Docker images and applying Terraform changes. I go into all the technical details of this process in the article on SaintsXCTF Kubernetes Infrastructure.
Other infrastructure changes include the use of API Gateway and AWS Lambda. There are two APIs deployed on API Gateway - an authentication API (auth.saintsxctf.com) and a function API (fn.saintsxctf.com). The authentication API has endpoints for creating authentication tokens and validating authentication tokens. These tokens are JWTs that have a lifespan of one hour. This authentication approach is a major improvement over my old implementation, which just used a static passcode in the authorization header for all API calls.
The function API serves many purposes, including profile picture uploads and all the email functionality on the website. Profile pictures are saved and accessed from an AWS S3 bucket.
SaintsXCTF has undergone massive changes since its initial release in December 2016. My knowledge of the software engineering landscape has grown a lot since my senior year of college, and that growth is displayed in the technological changes of SaintsXCTF. Learning from and evolving this application is invaluable to my growth as a software engineer, and I’m excited for its continued evolution in the future.