How to make friends AWS Lambda and PostgreSQL
Recently, the native support for Python 2.7 has appeared in the AWS Lambda service. For those who do not know what AWS Lambda is, I refer to the article on Habré . The main advantage of using the service is creating a scalable server-based architecture. In this article, I will discuss how to make friends with AWS Lambda and PostgreSQL.
The main problem of using AWS Lambda in conjunction with Python code arises from the fact that the environment running the code is a short-lived instance (virtual server) into which it is not possible to install the necessary binary components. Moreover, the use of python code requires the creation of a so-called deployment package. A typical scheme for creating such a package is briefly the following: copy the entire contents of $ VIRTUAL_ENV / lib / python2.7 / site-packages into the directory with the code used, zip the archive, download the console or command line utilities to the service via AWS, etc. P. I am interested in referring to the web service documentation, as I intend to write about a specific problem, rather than detailed settings.
If you try to use a similar scheme for working with sqlalchemy and PostgreSQL, then when you try to call code using a query to the database, you will receive something like the following:
This is due to the fact that psycopg2 - a library for working with PostgreSQL in Python - is an extension that uses C code and the shared library libpq.so from the PostgreSQL distribution.
')
The solution in this case is this (all actions were performed on the Fedora 22 distribution):
1) Downloading the PostgreSQL sources from http://www.postgresql.org/ftp/source
2) Execute commands
3) Create and activate virtualenv, which we use to create the deployment package:
4) Then we work with the psycopg2 repository:
5) From the src / interfaces / libpq subdirectory, copy the file libpq.a to the directory with the cloned psycopg2 distribution. Run the build:
Alternatively, put libpq.a in the / usr / lib64 directory or another, defined by the environment variable LD_LIBRARY_PATH, and uncomment the parameter static_libpq = 1 in the setup.cfg file.
4) Put the package:
Now using this virtualenv to create a deployment package for the absence of a shared library will not be abusive.
Unfortunately, this approach can not be called universal.In particular, I did not manage to solve the problem with the pillow. See UPDATE 2
I would be grateful if any of the habrazhiteli express constructive thoughts about this.
UPDATE. Here is an alternative solution:
1) When building a package, specify the rpath option as / var / task / lib:
In the deployment package, create a lib subdirectory where we put the binary shared library:
UPDATE 2. Pip with some version supports specifying build options in requirements.txt. So you can and so:
UPDATE 3. In the cloud, Amazon connects almost instantly, and during this year they promise the possibility of screwing the lambda to your VPC.
UPDATE 4. VPC is screwed, but the opportunity is gone connects to resources for VPC. Amazon is such an Amazon.
The main problem of using AWS Lambda in conjunction with Python code arises from the fact that the environment running the code is a short-lived instance (virtual server) into which it is not possible to install the necessary binary components. Moreover, the use of python code requires the creation of a so-called deployment package. A typical scheme for creating such a package is briefly the following: copy the entire contents of $ VIRTUAL_ENV / lib / python2.7 / site-packages into the directory with the code used, zip the archive, download the console or command line utilities to the service via AWS, etc. P. I am interested in referring to the web service documentation, as I intend to write about a specific problem, rather than detailed settings.
If you try to use a similar scheme for working with sqlalchemy and PostgreSQL, then when you try to call code using a query to the database, you will receive something like the following:
START RequestId: 23af180b-7b22-11e5-beac-0b062a6930ba Version: $LATEST Unable to import module 'lambda': libpq.so.5: cannot open shared object file: No such file or directory END RequestId: 23af180b-7b22-11e5-beac-0b062a6930ba REPORT RequestId: 23af180b-7b22-11e5-beac-0b062a6930ba Duration: 0.32 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 38 MB This is due to the fact that psycopg2 - a library for working with PostgreSQL in Python - is an extension that uses C code and the shared library libpq.so from the PostgreSQL distribution.
')
The solution in this case is this (all actions were performed on the Fedora 22 distribution):
1) Downloading the PostgreSQL sources from http://www.postgresql.org/ftp/source
2) Execute commands
tar xzvf postgresql-9.4.5.tar.gz cd postgresql-9.4.5 ./configure make
3) Create and activate virtualenv, which we use to create the deployment package:
virtualenv ./env source ./env/bin/activate
4) Then we work with the psycopg2 repository:
git clone -b maint_2_6 https://github.com/psycopg/psycopg2
5) From the src / interfaces / libpq subdirectory, copy the file libpq.a to the directory with the cloned psycopg2 distribution. Run the build:
cd psycopg2 python setup.py build_ext -L ./
Alternatively, put libpq.a in the / usr / lib64 directory or another, defined by the environment variable LD_LIBRARY_PATH, and uncomment the parameter static_libpq = 1 in the setup.cfg file.
4) Put the package:
python setup.py install
Now using this virtualenv to create a deployment package for the absence of a shared library will not be abusive.
Unfortunately, this approach can not be called universal.
I would be grateful if any of the habrazhiteli express constructive thoughts about this.
UPDATE. Here is an alternative solution:
1) When building a package, specify the rpath option as / var / task / lib:
python setup.py build_ext -R /var/task/lib
In the deployment package, create a lib subdirectory where we put the binary shared library:
(env)[random1st@localhost lambda]$ ls lib/ libpq.so libpq.so.5 libpq.so.5.7
UPDATE 2. Pip with some version supports specifying build options in requirements.txt. So you can and so:
(env)[random1st@localhost lambda]$ cat requirements.txt |grep option Pillow==2.5.1 --global-option="build_ext" --global-option="--rpath=/var/task/lib" psycopg2==2.5.3 --global-option="build_ext" --global-option="--rpath=/var/task/lib"
UPDATE 3. In the cloud, Amazon connects almost instantly, and during this year they promise the possibility of screwing the lambda to your VPC.
UPDATE 4. VPC is screwed, but the opportunity is gone connects to resources for VPC. Amazon is such an Amazon.
Source: https://habr.com/ru/post/269489/
All Articles