Monday, May 9, 2016

Lambda Environment Configuration with DynamoDB and /tmp


Currently AWS does not have a mechanism for Lambda deployment to encapsulate environment specific configuration.  You must include configuration files alongside your Lambda functional code.
When working on a photo architecture (https://runsignup.wordpress.com/2016/03/29/runsignup-photo-architecture/) we require ability to be a bit more dynamic in our approach for configuration and not depend on code updates.  We decided on using DynamoDB for managing this configuration, and improving performance by taking advantage of the /tmp storage available in container re-use as described here.   Below are the options we reviewed and the code for using Lambda + Dynamo + /tmp storage.

Here are some options for being more dynamic and code example for the approach we used.

[Note: If you are using API Gateway there is a better workaround, read here, we require S3 triggers]

a.  Update Lambda every time configuration changes and redeploy. 

Having configuration inside Lambda is a good from a performance perspective, but managing many customer specific Lambda's would be problematic.

b.  Use the Lambda description to encapsulate config object.

A hack that works (confirmed) and detailed here, however there is a  256 character limit for description field per documentation.

c. Use S3 to store the configuration file.

This allows remote storage of configuration via a file.  What's better Dynamo as document or S3 as a file?  Not too much, though the folks at Concurrency Labs had the same thought process and did a some performance measurements.  You can read more about their insights at their blog post here, the following is from their results.

d. Use a more dynamic approach with DynamoDB

Our approach was to use DynamoDB for the configuration store, while using /tmp to gain performance benefit when the container is re-used. 



1 comment:

travelling martian said...

actually u dont need to use /tmp if its a configuration file.
You can use global variable to hold it so that your evaluation is like this :
g_value = None

def xxxxx () :
global g_value
if not g_value :
# load the value from dynamo or s3
value = xxxxx
g_value = xxxxx
else :
value = g_value

This is even faster than relying on /tmp.

marcus@shopprapp.com