RETROSPECTIVE

February 5th, 2022

Building an API for Sending Emails with AWS Lambda and API Gateway

AWS Lambda

API Gateway

AWS

Terraform

JavaScript

Jenkins

HCL

s

This is part of a series of articles on SaintsXCTF Version 2.0. The first article in the series provides an overview of the application. You DO NOT need to read prior articles in the series to fully understand this article.

My SaintsXCTF application has multiple APIs. One of these APIs, fn.saintsxctf.com, is hosted on AWS API Gateway and AWS Lambda. Its main purpose is to send emails to users based on certain website actions, such as changing a password or registering. In this article, I discuss the infrastructure needed to make the API, the source code of the API endpoints, and the approach used to deploy the API to AWS.

The Infrastructure for my API that sends emails is written using Terraform and exists in my saints-xctf-infrastructure repository. The Terraform configuration contains three modules, two of which I'll discuss in this article: api-gateway and email-lambda. api-gateway contains API Gateway infrastructure, and email-lambda contains AWS Lambda infrastructure.

The following infrastructure diagram represents the fn.saintsxctf.com API.

This infrastructure creates a REST API that clients can access over HTTPS. For example, clients can make a POST request to the https://fn.saintsxctf.com/email/welcome endpoint to send a "welcome" email. API Gateway handles HTTP requests to endpoints and triggers corresponding AWS Lambda functions. Once an AWS Lambda function completes, its result is returned back to API Gateway and sent to the client in a HTTP response body.

Infrastructure for AWS API Gateway is centered around a main.tf file within the api-gateway Terraform module. Infrastructure for AWS Lambda is centered around a main.tf file within the email-lambda Terraform module. I won't discuss infrastructure specifics, but if you want to learn more about writing Terraform for AWS API Gateway and AWS Lambda I wrote an article about it in the past.

The API endpoints are configured as AWS Lambda functions written in JavaScript. All the Lambda functions exist in a saints-xctf-functions repository on GitHub.

Since most endpoints in the API are similar, let's look at a single endpoint, https://fn.saintsxctf.com/email/welcome, in detail. This endpoint sends a welcome email to new users when they initially sign up. A sample welcome email is displayed below.

Lambda function code for the welcome email endpoint exists in an npm module. The module has two entrypoint files: sendEmailAWS.js and sendEmailNode.js. sendEmailAWS.js is used for sending emails from AWS Lambda and sendEmailNode.js is used for sending emails locally.

// sendEmailAWS.js const AWS = require('aws-sdk'); const sendWelcomeEmail = require('./email'); AWS.config.update({region: 'us-east-1'}); /** * Invoke an AWS function via the handler object * @param event - information about the AWS Lambda function request * @param context - runtime information regarding the AWS lambda function * @param callback - used to return information back to the caller. */ exports.sendWelcomeEmail = (event, context, callback) => { try { sendWelcomeEmail(event.to, event.firstName, event.lastName, event.username); callback(null, true); } catch (err) { console.error(err); callback(Error(err), false); } };
// sendEmailNode.js const sendWelcomeEmail = require('./email'); const [_, __, to, firstName, lastName, username] = process.argv; sendWelcomeEmail(to, firstName, lastName, username);

sendEmailNode.js is great for testing. It makes sending an email as simple as running the following command from the command line.

node sendEmailNode andrew@jarombek.com Andrew Jarombek andy

In sendEmailAWS.js, exports.sendWelcomeEmail() is the main function. In the Lambda function's Terraform configuration, exports.sendWelcomeEmail is configured as the entrypoint to the AWS Lambda function using the handler attribute.

In sendWelcomeEmail(), the event argument holds values sent by users invoking the API. These values are sent via a JSON request body. Values include the recipient's email address (to) and the first name (firstName), last name (lastName), and username (username) of the new user. to is the address that receives the email and the other three fields replace values in a templated email body (which I discuss soon).

Both sendEmailNode.js and sendEmailAWS.js invoke a sendWelcomeEmail() function. sendWelcomeEmail() is imported from a ./email.js file with the line const sendWelcomeEmail = require('./email'). This file is shown below.

// email.js const send = require('aws-lambda-emails'); /** * Send a welcome email to a new user. * @param to Recipient email address. * @param firstName First name of the new user. * @param lastName Last name of the new user. * @param username Unique username of the new user. */ function sendWelcomeEmail(to, firstName, lastName, username) { const subject = "Welcome to SaintsXCTF"; const attachments = [ { filename: 'saintsxctf.png', path: './assets/saintsxctf.png', cid: 'saintsxctf@jarombek.com' } ]; const htmlFilename="email"; const cssFilename="email"; const replacementValues = { firstName, lastName, username, prefix: process.env.PREFIX || '' }; send(subject, to, attachments, htmlFilename, cssFilename, replacementValues); } module.exports = sendWelcomeEmail;

sendWelcomeEmail() invokes a send() method from an aws-lambda-emails npm module I created. The send() method takes care of sending the email, but takes parameters to configure how the email is composed.

send() is invoked with the line send(subject, to, attachments, htmlFilename, cssFilename, replacementValues). subject is the title of the email and to is the recipients email address. attachments is a list of attachments for the email. In the case of the welcome email, there is a single saintsxctf.png image attachment. htmlFilename and cssFilename are the names of the HTML and CSS files that form the email body. In the welcome email code, both these variables have a value of email, which references email.html and email.css files in my repository. email.html is shown below.

<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Welcome to SaintsXCTF</title> </head> <body bgcolor="#f6f8f1"> <table class="emailBody" width="100%" bgcolor="#f6f8f1" border="0" cellpadding="0" cellspacing="0"> <tr> <td> <div class="emailContent"> <table class="emailMain" width="100%" align="center" border="0" cellpadding="10" cellspacing="10"> <tr> <td> <table border="0" cellspacing="0" cellpadding="0"> <tr> <td> <table class="emailHeader" border="0" cellpadding="0" cellspacing="0"> <tbody> <tr> <td> <table border="0" cellpadding="0" cellspacing="0"> <tbody> <tr> <td width="80" height="80"> <img class="emailLogo" src="cid:saintsxctf@jarombek.com" alt=""/> </td> <td> <p class="emailTitle"> SaintsXCTF </p> </td> </tr> </tbody> </table> </td> </tr> </tbody> </table> <p class="emailh5"> Welcome to SaintsXCTF </p> <p class="emailp"> Hello {{firstName}} {{lastName}}, </p> <p class="emailp"> Welcome to SaintsXCTF, an application for logging exercises with your Cross Country and Track & Field teams. Your username is <b>{{username}}</b>. </p> <p class="emailp"> If you experience any issues using the website, please contact Andrew Jarombek at <b>andrew@jarombek.com</b> (or if you are currently a St. Lawrence athlete, contact a member of the coaching staff). I won't be able to fix issues if I don't know they exist, so your help is appreciated! </p> <p class="emailp"> I hope you enjoy the website and have a great journey at St. Lawrence University! </p> <p class="emailp"> - Andrew Jarombek (Website Creator & Class of 2017) </p> <table class="buttonContainer" align="center" border="0" cellpadding="0" cellspacing="0"> <tbody> <tr> <td align="center"> <table border="0" cellpadding="0" cellspacing="0"> <tbody> <tr> <td class="button" height="40" width="250"> <a href="https://{{prefix}}saintsxctf.com"> BEGIN LOGGING EXERCISES </a> </td> </tr> </tbody> </table> </td> </tr> </tbody> </table> </td> </tr> </table> </td> </tr> </table> </div> </td> </tr> </table> </body> </html>

Take note of words in the email surrounded by double curly braces, such as {{firstName}}, {{lastName}}, and {{username}}. These are template variables that are replaced with values when the email is rendered. This is where the final parameter of send(), replacementValues, comes into play. replacementValues is an object where the field names are the template variables (such as {{firstName}}) and the field values are the replacement values (such as Andy). This approach allows me to create personalized emails for users of the SaintsXCTF website.

The last piece of the puzzle is the send() function in the aws-lambda-emails module. send() exists in a index.js file.

const AWS = require('aws-sdk'); const nodemailer = require('nodemailer'); const fs = require('fs'); const juice = require('juice'); /** * Main function to configure and send an email. * @param subject The subject line of the email. * @param to Who the email is sent to. * @param attachments Email attachment files. * @param htmlFilename Name of the html file to use for the email body. * @param cssFilename Name of the css file used to style the email body. * @param replacementValues Values in the HTML file to replace. The name of the properties in the * object matches values in the html to replace. These properties values are the * replacement values. */ function send(subject, to="andrew@jarombek.com", attachments = [], htmlFilename="email", cssFilename="email", replacementValues = {}) { // Read the contents of the HTML and CSS files to send with the email let html = fs.readFileSync(`./${htmlFilename}.html`, 'utf8'); const css = fs.readFileSync(`./${cssFilename}.css`, 'utf8'); // Replace the templates in the HTML for (const key in replacementValues) { html = replace(html, key, replacementValues[key]); } // Inline the CSS styles in the HTML document const styledHtml = juice.inlineContent(html, css); const secretsManager = new AWS.SecretsManager({ region: 'us-east-1' }); secretsManager.getSecretValue({ SecretId: 'google-account-secret' }, (err, data) => { if (err) { console.error(err); } else { const { SecretString } = data; const secretObject = JSON.parse(SecretString); const password = secretObject.password; const transport = createTransport(password); sendMail(transport, subject, to, attachments, styledHtml); } }); } /** * Configure how the email is transported. * @param password Application specific Gmail password used to send emails in a secure manner. */ function createTransport(password) { return nodemailer.createTransport({ host: 'smtp.gmail.com', port: 465, secure: true, auth: { user: 'andrew@jarombek.com', pass: password } }); } /** * Helper method to send the email and check for errors. * @param transport Transport object used to send emails. * @param subject The subject line of the email. * @param to Who the email is sent to. * @param attachments Email attachment files. * @param html Styled HTML file used as the email body. */ function sendMail(transport, subject, to, attachments, html) { transport.sendMail({ from: 'Andrew Jarombek<andrew@jarombek.com>', to, subject, html, attachments }, (err) => { if (err) { console.error(`Something went wrong sending mail: ${err}`); } }); } /** * Replace a template item in a string. Templates are denoted by the {{<template>}} pattern * @param string -The string with the template items * @param substring The <template> identifier * @param replacement String that will replace the template item * @return {Object|string|void|*} A new string - note strings are immutable so a new string * object will be made. */ function replace(string, substring, replacement) { return string.replace(`{{${substring}}}`, replacement); } module.exports = send;

I included send() and all its helper functions in the code above. At its core, the code uses a nodemailer module to send emails. nodemailer.createTransport() configures the transportation layer of the email by authenticating with my Gmail account. My Gmail account password is stored on AWS Secrets Manager and is retrieved with the secretsManager.getSecretValue({ SecretId: 'google-account-secret' }) function call. It's important to note that for this code to work for you, you also need an AWS account with a secret of the same name. For local development purposes only, you can hardcode the account password instead.

transport.sendMail({...}) sends an email based on the arguments passed to send(). Before this method is called, the email body is configured and styled within send(). The following few lines from send() generates the email body:

// Read the contents of the HTML and CSS files to send with the email let html = fs.readFileSync(`./${htmlFilename}.html`, 'utf8'); const css = fs.readFileSync(`./${cssFilename}.css`, 'utf8'); // Replace the templates in the HTML for (const key in replacementValues) { html = replace(html, key, replacementValues[key]); } // Inline the CSS styles in the HTML document const styledHtml = juice.inlineContent(html, css);

At the top of send(), Node.js' fs module reads the email's HTML and CSS files into html and css variables, respectively. I also iterate through all the key-value pairs in replacementValues, replacing the templates with their designated values in the HTML. This is accomplished with the help of my replace() function.

juice.inlineContent(html, css) uses the juice library to inline CSS styles in HTML elements1. The result, styledHtml, is the HTML sent in the email body.

The approach for deploying the API and its corresponding AWS Lambda functions is a CI/CD pipeline in the form of a Jenkins job. Jenkins is a CI/CD platform that I've discussed many times in the past. Jenkins jobs are pipelines written in Groovy that automate tasks such as deploying an application to a specific environment or unit testing source code.

The Jenkins job for my API contains two general steps: creating AWS Lambda Zip files and applying Terraform infrastructure. The Jenkinsfile for the Jenkins job, shown below, utilizes some helper functions from my global-jenkins-library repository.

@Library(['global-jenkins-library@master']) _ pipeline { agent { label 'master' } parameters { booleanParam( name: 'autoApply', defaultValue: true, description: "Whether the Terraform infrastructure should be automatically approved." ) choice( name: 'environment', choices: ['dev', 'prod'], description: 'Environment to build the infrastructure in.' ) } options { ansiColor('xterm') timeout(time: 1, unit: 'HOURS') buildDiscarder(logRotator(daysToKeepStr: '10', numToKeepStr: '5')) } stages { stage("Clean Workspace") { steps { script { cleanWs() } } } stage("Checkout Repositories") { steps { script { checkoutRepos() } } } stage("Get Lambda Zip Files") { steps { script { createLambdaZipFiles() } } } stage("Terraform Init") { steps { script { terraformInit() } } } stage("Terraform Validate") { steps { script { terraformValidate() } } } stage("Terraform Plan") { steps { script { terraformPlan() } } } stage("Terraform Apply") { when { allOf { environment name: 'TERRAFORM_NO_CHANGES', value: 'false' environment name: 'TERRAFORM_PLAN_ERRORS', value: 'false' } } steps { script { terraformApply() } } } } post { always { script { postScript() } } } } def checkoutRepos() { genericsteps.checkoutRepo('saints-xctf-infrastructure', 'master') genericsteps.checkoutRepo('saints-xctf-functions', 'main') } def createLambdaZipFiles() { dir('repos/saints-xctf-functions/forgot-password') { sh """ yarn install --production=true zip -r9 SaintsXCTFForgotPasswordEmail.zip . cp SaintsXCTFForgotPasswordEmail.zip ../../saints-xctf-infrastructure/saints-xctf-com-fn/modules/email-lambda """ } dir('repos/saints-xctf-functions/send-activation') { sh """ yarn install --production=true zip -r9 SaintsXCTFActivationCodeEmail.zip . cp SaintsXCTFActivationCodeEmail.zip ../../saints-xctf-infrastructure/saints-xctf-com-fn/modules/email-lambda """ } dir('repos/saints-xctf-functions/report') { sh """ yarn install --production=true zip -r9 SaintsXCTFReportEmail.zip . cp SaintsXCTFReportEmail.zip ../../saints-xctf-infrastructure/saints-xctf-com-fn/modules/email-lambda """ } dir('repos/saints-xctf-functions/welcome') { sh """ yarn install --production=true zip -r9 SaintsXCTFWelcomeEmail.zip . cp SaintsXCTFWelcomeEmail.zip ../../saints-xctf-infrastructure/saints-xctf-com-fn/modules/email-lambda """ } dir('repos/saints-xctf-functions/upload-profile-picture') { sh """ yarn install --production=true zip -r9 SaintsXCTFUassetUser.zip . cp SaintsXCTFUassetUser.zip ../../saints-xctf-infrastructure/saints-xctf-com-fn/modules/uasset-lambda """ } dir('repos/saints-xctf-functions/upload-group-picture') { sh """ yarn install --production=true zip -r9 SaintsXCTFUassetGroup.zip . cp SaintsXCTFUassetGroup.zip ../../saints-xctf-infrastructure/saints-xctf-com-fn/modules/uasset-lambda """ } } def terraformInit() { INFRA_DIR = "repos/saints-xctf-infrastructure/saints-xctf-com-fn/env/$params.environment" terraform.terraformInit(INFRA_DIR) } def terraformValidate() { terraform.terraformValidate(INFRA_DIR) } def terraformPlan() { terraform.terraformPlan(INFRA_DIR) } def terraformApply() { terraform.terraformApply(INFRA_DIR, params.autoApply) } def postScript() { email.sendEmail( "Create ${params.environment.toUpperCase()} SaintsXCTF Functions AWS Infrastructure", "", env.JOB_NAME, currentBuild.result, env.BUILD_NUMBER, env.BUILD_URL ) cleanWs() }

Note that this pipeline is able to deploy infrastructure to development (dev) and production (prod) environments. Running this pipeline is the only step needed to deploy a new version of the API to AWS.

This approach of building an API with AWS API Gateway and AWS Lambda is great for small APIs with a few endpoints. These API endpoints that send emails are a good fit for AWS Lambda because they run infrequently and are not very time sensitive. Due to the nature of AWS Lambda function cold starts, these endpoints sometimes take over a second to execute, which would be unacceptable for more performance-critical endpoints. For more complex APIs written with AWS Lambda, I'd recommend trying a more sophisticated framework such as serverless. The code discussed in this article is available in saints-xctf-functions, saints-xctf-infrastructure, aws-lambda-emails, and global-jenkins-jobs repositories.