How to work from AWS Lambda with Elasticache and DynamoDB

Since February 2016, AWS Lambda has been able to access resources inside the Virtual Private Cloud , but by default all lambdas work outside of VPC. Since this opportunity has appeared relatively recently and there are not so many articles on how to implement it, we would like to share our experience with you.

We have developed a mobile application built on AWS. The Lambda service was great for writing a small server logic, moreover, it is easily scalable.

The lambda task was posed rather trivial: perform a query to DynamoDB and cache the most popular queries using Elasticache. The problem we faced is related to setting up access from VPC to internal and external Amazon resources.
')
In our case, it was necessary to ensure the availability of the Elasticache service (in VPC) and DynamoDB (outside VPC). To achieve this goal, a structure has been deployed in VPC, which is represented in the diagram:



The following is an instruction on how to create such a VPC operation scheme from the AWS Console.

VPC Setup
First you need to create a VPC, if you have not already done so. Create two subnets (Subnets) public and private.

The second step is to create a routing table (Route Tables). By default, you have already created one main table, it will serve as a private table. It remains only to create a public table. When creating, you must specify our VPC.

For each subnet (public and private), select the corresponding routing table.

Next, we create an internet gateway that will provide access to all services outside the VPC and to external addresses as a whole. After attaching the created gateway to our VPC. We also create a NAT gateway, which we attach to the public subnet.

And after all it remains only to configure the routing table:

• Create a route in a public table for the created Internet gateway. Example:

• Create a route in a private table for NAT. Example:

Elasticache setup

In our example, we consider connecting Elasticache to a private subnet. To do this, in the Elasticache panel, create a group for the private subnet (Cache Subnet Group). And then when creating a cluster, we indicate the created group.

After all, it remains only to find the group in the Security Groups for our VPC in the VPC panel (it is created automatically) and add a rule for the port of our cluster (the default redis is 6379; memcached is 11211). Example:



Let's check how it all goes with us. In the lambda settings, select VPC and private network. Run the code:

from __future__ import print_function import json import boto3 import decimal import logging import elasticache_auto_discovery from pymemcache.client.hash import HashClient CACHE_ENDPOINTS = 'exaplecache.wjbxgg.cfg.euw1.cache.amazonaws.com:11211' def decimal_default(obj): if isinstance(obj, decimal.Decimal): return float(obj) raise TypeError def lambda_handler(event, context): nodes_endpoints = elasticache_auto_discovery.discover(CACHE_ENDPOINTS) nodes = map(lambda x: (x[1], int(x[2])), nodes_endpoints) memcache_client = HashClient(nodes, timeout=60, connect_timeout=2) table = boto3.resource('dynamodb').Table('Photo') response = table.scan(Limit=20) memcache_client.set('example', json.dumps(response['Items'], default=decimal_default)) cache_elements = memcache_client.get('example') if json.loads(cache_elements) == response['Items']: return "It works!" else: return "Doesn't work" 

And after the launch we will see the coveted result:


Conclusion
Working with Amazon services inside and outside the VPC opens up many possibilities for using AWS Lambda. The configuration scheme for VPC presented in this article can be used as a universal solution to provide access to any other services inside and outside the VPC.

Useful links:
Lambda in vpc
Your VPC and Subnets
Getting Started with Amazon ElastiCache