I recently shared Foghorn’s AWS Hackaton for Social Good experience on our blog, and promised some deeper explanation of the technical details. To recap- we had secured our repeat title win by partnering with GameChanger, a company that leverages gaming to engage and inspire children suffering from life-threatening illnesses.  Once we understood their imminent needs (a test harness for their integration to Twitch), we decided to enhance the experience. We wanted a feature that could help increase their donations and make it easier for donors to engage with each other. To do so, we thought up the idea of making a mosaic composed of the donors’ avatars. Every time someone donated, it recreated the mosaic with each of the donor’s image embedded into it. Now how to build it…

After creating an architecture, we broke mosaic sub-project into 3 tasks:

• Automate getting the avatars into Amazon S3
• A trigger that called a lambda when a new image was uploaded to Amazon S3
• A script to run to generate the mosaic image of the avatars

The first task was relatively easy – we just needed to modify GameChanger’s Lambda. The Lambda code they provided was taking a notification from Amazon Pay (containing information about the Twitch user) and sent a Twitch notification to post saying `${user_name} donated $${amount}!` for everyone to see. We added a function which downloaded the user’s Twitch avatar into Amazon S3. The hardest part about this build was ensuring that we put the object in the correct prefix. We designed the bucket to handle N number of events, with the prefix being event name.

The second task we took on was to generate the mosaic image.  Here, we ran into some challenges. The issues included:

• Getting the files from s3
• Uploading Libraries
• Python Queues not supported

We found python code online that was a good starting point. This code took a given image and a folder of other images and parallel processed them into a mosaic. As I was in charge of converting the code we found into a lambda, I decided the first step was to make a wrapper for the script and then automate the input paths.

It would have to automatically parse the S3 Event notification and get the original image and a path to all the avatars. However, since I wanted to make this scale, I encouraged everyone to use the AWS CLI, which from past experience, is easier and multithreaded by default compared to boto3.  I assumed the AWS CLI was installed on AWS Lambda and once I tested on my local machine, I was ready to upload the code to AWS Lambda.

As many are aware, when uploading AWS Lambda function with custom libraries, it requires that those libraries are included. This was the case here since we were using the python Pillow library. Typically, I use AWS recommended solution of a virtual environment and then zip up everything. I have never preferred this process due to the iteration time of this and therefore have tried to use different techniques in bash scripts or terraform to decrease the cycles. However, my teammate recommended py-lambda-packer, which makes packaging much easier and just by configuring 1 YAML file. Unfortunately, here is where we started to run into problems. After packaging the lambda and uploading it to AWS, we were seeing the following error: “Unable to import module ‘lambda’: cannot import name ‘_imaging’.” I had checked to ensure Pillow was included in my requirements.txt file. After many google searches, it turns out that Pillow has to be in root lambda zip. Therefore, the workaround is to copy the PIL folder from a pip install and include that in the ZIP file with the lambda code.

Example of py-lambda-packer

virtualenv:
python: python3.6
pip:

requirements:
– requirements.txt

packager:
target: lambda_mosaic.zip
followlinks: true
includes:
– lambda.py
– PIL
– aws

We followed this workaround to try to resolve the issue.  At this point, given that we had no “startup” errors on lambda, we thought we would have no more issues (especially since everything ran on my laptop). Needless to say, that was not the case. We found that the AWS CLI was not installed (like I originally assumed) in the AWS Lambda container. I attempted to include the AWS executable and all corresponding packages, but it never worked as expected. To ensure we kept moving, I ended up reverting back to boto3 with a note in the code saying this was non-multi threaded and if we are doing this at a large scale we might want to fix this in the future.

Once we successfully downloaded the input image and all the avatars that made up the output image, we needed to execute the code mosaic code we found online. When running the AWS Lambda, we saw the following: “OSError: [Errno 38] Function not implemented” error. After about 45 minutes of debugging with a teammate, we came across this AWS forum. In short, it outlines that AWS Lambda does not support the version of multithreading that uses shared memory. At this point, we had 3 hours left and had distilled our options down to the 3 below:

• Rewrite the threading in a supported method
• Remove the threading and make it single threaded
• Stop trying to make it run in lambda, rearchitect it to run on containers

Since both of us use containers daily at work, we choose option 3.  We re-architectured the process to have the S3 notification go through an SQS queue and then have the container read the SQS queue. We built the container quickly as a proof of concept.

FROM ubuntu:xenial
RUN apt-get update && apt-get install -y python-pip gcc python3-pip python3
COPY . /app
WORKDIR /app
RUN pip3 install -r requirements.txt
ENTRYPOINT [“python3”]
CMD [“process.py”]

After a long day of many roadblocks, the code/process finally worked. As a team, we were happy to hand GameChanger working code they can implement relatively easily. We hope that our work will allow GameChanger to continue to make an impact and encourage more people to donate to their amazing cause.

Are you currently building serverless architectures on AWS? Give us a ring, maybe we can help!