If you have created website-based applications for any length of time, you are familiar with the pitfalls of deploying your application to production. Never are you more at risk of an outage than when you begin to install your website application in production. To address this risk, the concept of continuous delivery was created, and the idea of continuous delivery is to release your application frequently, rapidly, and automatically, and to provide a mechanism to quickly roll back a release should it have a negative impact on your user experience.
One technique to release software using the continuous delivery philosophy is that of blue green deployment. Blue Green deployment is a technique of releasing applications in which you build two production environments as identical as possible, and then you quickly swap traffic over from one to the other. A major advantage of blue green is that no matter how complicated it is to install and configure your application software, doing so has zero impact on your application’s users until you switch the traffic over to use the new version.
To see how blue green works, take a look at the following simplified design:
1. A blue instance of your application running one version
a. The color blue is chosen to represent the concept of frozen
b. This infers that the version of your application running in this instance should not be modified
2. A green instance of your application running a different version
a. The color green is chosen to represent new growth, like a green field
b. This infers that the version of your application running in this instance is new and in the process of being changed and tested
3. A router
a. This element uses some intelligence about the traffic it receives to route the traffic to either the blue version or the green version of your application
b. Routing can be done by any part of the HTTP request, but typically is done by the domain name or an HTTP header
4. Live and test URLs
a. These elements of the application bifurcate users
b. Normal, regular users of your application would use the live URL
c. Users engaged in testing your new application version would use the test URL
d. Users never have to change which URL they use, the router handles switching their request to either the blue or green instance of your application
One type of modern website application you could build is an AWS Cloudfront site with an S3 origin. This type of website application allows you to host a serverless application, using static website content stored in an S3 bucket, protected with an AWS ACM Certificate, and cached regionally close to a global user base. For example, see the architectural diagram below:
In order to implement blue green deployment on a website application such as this, you would create blue and green instances of your S3 bucket content, perhaps in blue and green folders within the bucket. Additionally, you would create another Route53 record for your website application and point it to your Cloudfront so that your test users could access the “green” application. But, how would you configure Cloudfront so that it takes on the role of “Router”?
AWS provides Lambda@Edge for this exact purpose. You can use Lambda@Edge functions to change Cloudfront requests and responses, such as changing the S3 origin used by Cloudfront so that it will serve your content from a blue or green path. Cloudfront supports changing the request or response either pre or post cache, as shown in the diagram below:
The functions that change a request are referred to as a Viewer request and Origin request, and the functions that change a response are referred to as Viewer response and Origin response. Both “Viewer” functions operate on the request/response outside of what is cached, and this is a very important property to take into account for the purposes of configuring Cloudfront to be your blue green Router. You want your Router to instantly switch your Live traffic from blue to green, and you do not want your Live users getting served cached content after the blue green switch occurs. Therefore you need to use a Viewer request Lambda@Edge function to serve as your router, so that it can make this switching decision before Cloudfront determines if your request has a cache hit.
However, the Viewer request Lambda@Edge function does not have access to any details of the origin server, in this case our S3 bucket. So you must also use an Origin request Lambda@Edge function to dynamically choose the S3 origin based on some determination made by the Viewer request Lambda@Edge function.
Another important behavior to note is that Origin requests are only made by Cloudfront if the request from the user does not find a cache hit. Therefore it is also critical for your blue green Router configuration to make sure the defined Cloudfront caching behavior includes some aspect of the HTTP request that can be switchable. One technique you can use to ensure your user requests will cache properly is by dynamically inserting a new HTTP header into the request that indicates whether or not the traffic should be served by the blue or green instance of the application, something like x-blue-green-context, and then configuring the Cloudfront caching behavior to include that header as part of the cache determination.
So now that we know how Lambda@Edge functions work to change requests, let’s put all of our knowledge together into a working Viewer request function and Origin request function. First, we will look at our Viewer request function:
def lambda_handler(event, context):
request = event['Records'][0]['cf']['request']
headers = request['headers']
hostName = headers.get('host')[0]['value']
subDomain = hostName.split('.')[0]
print("Adding blue green header for host %s" % hostName)
if subDomain.endswith('-test'):
request['headers']['x-blue-green-context'] = [{'key': 'x-blue-green-context', 'value': "green"}]
else:
request['headers']['x-blue-green-context'] = [{'key': 'x-blue-green-context', 'value': "blue"}]
return request
In this Viewer request function, we use the host name of our Route53 record to determine whether the x-blue-green-context header should switch the traffic to the blue S3 bucket folder or the green folder. In this implementation, we simply look to see if the subdomain ends with the text -test to send our test users to the green folder, otherwise they get sent to the blue folder. For this Lambda@Edge function, we only need to indicate which folder to use, thus we simply set the x-blue-green-context header to a value of blue or green, and we will let this header pass through to the Origin request function for it to dynamically switch the S3 origin.
It should also be noted here that the x-blue-green-context header being switched from blue to green for your live traffic at some point will cause any cache for your live traffic to miss, and thus your Cloudfront will begin to make new origin requests for your live traffic instead of serving cached responses.
Now we can look at our Origin request function:
def lambda_handler(event, context):
request = event['Records'][0]['cf']['request']
headers = request['headers']
s3DomainName = headers.get('host')[0]['value']
s3Region = s3DomainName.split('.')[2]
contextHeader = headers.get('x-blue-green-context')
if contextHeader:
blueGreenContext = contextHeader[0]['value']
blueGreenPath = "/%s" % blueGreenContext
print("Blue Green Context is %s" % blueGreenContext)
request['origin'] = {
's3': {
'domainName': s3DomainName,
'region': s3Region,
'authMethod': 'origin-access-identity',
'path': blueGreenPath,
'customHeaders': {}
}
}
else:
print("Did not find Blue Green Context")
return request
Here you can see the Origin request function fetches the x-blue-green-context header from the HTTP request, that was inserted by the Viewer request function, and uses its value to change the configuration of the request’s origin server, specifically the path attribute of the S3 origin. If the traffic originally came from our test user’s Route53 domain, the Origin request function will set the S3 path to the header’s value which would be green. Otherwise, if the traffic originally came from the live user’s Route53 domain, the Origin request function will set the S3 path to the header’s value which would be blue. In either case, the Origin request function does not determine whether the traffic should go to the blue folder of the S3 bucket or the green folder, it just implements the decision that was made.
So how do we switch the traffic? We simply update the Viewer request Lambda@Edge function so that test traffic gets a header value of blue instead of green, and live traffic gets a header value of green instead of blue. Once we make that change, we deploy the new function code and then update Cloudfront to use the new function version. This last step is critical as Lambda@Edge functions run on the edge, that is regionally close to your global user base, and therefore AWS must push that new function’s code out to all edges (cloudfront regions in use).
If you would like to see this fully functional for a simple site, we have a sandbox implementation you can use at https://github.com/sbwise01/mylambdaedge. This sandbox contains Hashicorp Terraform configuration that you can use to provision all of the necessary resources in an AWS account to demo this functionality, including the S3 website content.
Stay tuned for future blog posts on blue green deployments for other types of websites. And feel free to contact Foghorn Consulting to assist you with building continuous delivery automation using the blue green deployment technique for your web applications.
