Creating a CI/CD Pipeline with CodePipeline, CodeBuild, RDS and Route53

CodePipeline is a managed product that can be used to create an automated software release process designed to catch bugs and defects quickly and prevent them entering a production environment.

In this article we'll be looking at how CodePipeline can be used in conjunction with the rest of the AWS Code suite in addition to integrations with Route53, RDS and Secrets Manager.

What We'll Be Doing

By the end of this article, we'll have covered the following components of our pipeline:

• A webhook that triggers a new deployment whenever a "push" is made on a GitHub repository
• Miscellaneous build-related secrets stored in Secrets Manager
• A build script that provisions a temporary database which we'll be using for integration tests. This will include setting up a predefined CNAME through Route53
• A teardown script that decommissions the temporary database
• A buildspec.yml file which sets environment variables through Secrets Manager, installs dependencies, runs unit and integration tests, and performs any necessary cleanup.

Note For this example, we will be assuming the tests are run against a monolithic Ruby on Rails application, which means there will be some terminology specific to Ruby on Rails in the sections that follow.

A Few Important Points

This article assumes a basic familarity with CI/CD principles and a working knowledge of AWS services.

We also assume a basic level of familarity with Docker and the ability to push a Docker image to the Elastic Container Registry.

Finally, we will not be covering the writing of the tests themselves. Instead we will be focusing on the automation of those tests during the deployment process.

Disclaimer The architecture described here is not necessarily the most cost-effective or efficient; the article is just designed to explore how different cloud services can be combined together to solve problems. Moreover, it would normally be better to use Infrastructure as Code to define this kind of infrastructure configuration, but for the sake of speed we will not be doing that here.

Let's Set up the Pipeline

Let's get started! The first place to start is Secrets Manager, where we'll be adding some sensitive values that will be needed during the build process.

Creating our Secrets

We will need to create two secrets to store sensitive values. One of these is for the temporary database's parameters like username and password and the other is for the CodeBuild environment.

First we create the TestDB secret:

Next, we can add the values it stores:

• TEST_DB_CNAME
• HOSTED_ZONE_FOR_TEST_DB
• TEST_DB_NAME
• TEST_DB_USERNAME
• TEST_DB_PASSWORD
• TEST_DB_SECURITY_GROUP

TEST_DB_SECURITY_GROUP refers to the name of a security group that will be attached to the temporary test database. This security group should be configured in advance.

TEST_DB_CNAME refers to a CNAME that will point to the public DNS hostname of the temporary database. The subdomain will need to be part of a domain that you own and which is managed through Route53.

HOSTED_ZONE_FOR_TEST_DB is the Route53 Hosted Zone ID in which the domain's records are managed.

The other three values will hopefully be self-explanatory and you can set them as you see fit.

Finally, we have the option to configure rotation of the key used to back the secret:

We will also need to repeat this for the CodeBuildEnvVariables secret which contains the following values:

• DATABASE_URL
• DEVISE_JWT_SECRET_KEY, which is used to configure the Rails authentication layer

Setting up CodePipeline

Now that the secrets have been created, we can start building the pipeline.

First, let's create a pipeline.

Next, we need to choose a source integration with a version control system. In the example, I have used GitHub (Version 2).

Now, we need to create an AWS CodeBuild project.

The final step is to configure the build project. As mentioned previously, you might need to use a custom Docker image to run your build, which will require familiarity with both Docker and Elastic Container Registry. If you are using Elastic Container Registry, there will need to be an IAM role attached to the build which has sufficient permissions to pull images from it.

Most of the other boxes will be filled in by the later steps in this article, so stay tuned.

Note There is a final CodePipeline stage which is the CodeDeploy integration. As this article assumes a Rails application, this is likely to require either an integration with Elastic Beanstalk or Elastic Container Service, both of which are relatively painless to set up. The CodeDeploy stage is not covered in this article because it is focused more on the build.

Route53

Now that our CodeBuild configuration is nearly complete, we need to turn our attention to provisioning a temporary database to run our tests.

To do this, we'll need the database to be available at a predictable URL or else our application code will need to change every time we deploy.

In order to complete this part, a domain is required. If the domain is not managed in Route53, it will need to be transferred.

Temporary Database

In an ideal world we could have our testing database operational all the time, but to keep costs down it would be better to only provision the database when it's required, even if that comes at the expense of longer CodeBuild execution times.

What this means in practice is that we will need to add a shell script which provisions the temporary database using various AWS CLI commands spanning a number of different services.

Build Script

Let's add a shell script to provision our temporary database.

This script does the following:

• Extracts values from the Secrets Manager secret that we configured earlier
• Extracts the ID of the pre-existing test database security group
• Creates an RDS database instance using the credentials extracted from the Secrets Manager secret
• Runs a command to pause script execution until the database is operational
• Adds a Route53 CNAME record using the public DNS hostname of the database which is extracted from a CLI command issued to the RDS API

Cleanup Script

Now we need to add a shell script responsible for tearing down the temporary database. This script will be run whether the CodeBuild execution is successful (meaning the integration tests pass) or unsuccessful (meaning we have some failing tests).

This script does the following:

• Extracts values from the Secrets Manager secret
• Deletes the temporary database instance, including any automated backups
• Removes the CNAME record from Route53

Putting It All Together

Now that we have got these two shell scripts which will provision and decommission the temporary testing database as required, we can move on with the codebuild.yml, which acts as a series of instructions for CodeBuild:

Note that the env, or environment, references values we previously defined in Secrets Manager. Note also how the cleanup shell script will be run in case of success or failure, ensuring that the temporary database is always decommissioned no matter what the outcome of the testing might be.

Note Because the shell scripts issue AWS CLI commands, the IAM service role attached to the CodeBuild environment will require - in addition to the ECR permissions we discussed earlier - read and write access to RDS, Route53, VPC Security Groups and Secrets Manger.

Assuming the Pipeline is now fully configured along with the CodeDeploy stage, you should now be able to see the CodeBuild process in action next time you push to the repository.

Summary

This article was a quick guide to demonstrate how Secrets Manager, AWS CodePipeline, AWS CodeBuild, RDS and Route53 can be used to create a deployment pipeline providing a quality-controlled release process which ensures high quality by running automated integration tests against a temporary testing database.