(aws_lambda_python): Poetry exported requirements not readable by pip
See original GitHub issuePip refuses to install the requirements generated by poetry export
.
Reproduction Steps
- Create a simple pyproject.toml and poetry.lock in a directory with existing Lambda job code (
poetry init && poetry add jsonschema="*" pynamodb="*"
) - Instantiate a PythonFunction for the directory above
cdk synth
What did you expect to happen?
The build should install the packages mentioned in poetry.lock.
What actually happened?
Step 1/9 : ARG IMAGE=amazon/aws-sam-cli-build-image-python3.7
Step 2/9 : FROM $IMAGE
---> 8aae4ac71397
Step 3/9 : RUN yum -q list installed rsync &>/dev/null || yum install -y rsync
---> Using cache
---> 97cffc20d856
Step 4/9 : RUN pip install --upgrade pip
---> Running in 71bfdf9b78cb
Collecting pip
Downloading pip-21.0.1-py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.2.3
Uninstalling pip-20.2.3:
Successfully uninstalled pip-20.2.3
Successfully installed pip-21.0.1
Removing intermediate container 71bfdf9b78cb
---> 4509190dda1a
Step 5/9 : RUN pip install pipenv poetry
---> Running in fd0a1868ac36
Collecting pipenv
Downloading pipenv-2020.11.15-py2.py3-none-any.whl (3.9 MB)
Collecting poetry
Downloading poetry-1.1.6-py2.py3-none-any.whl (172 kB)
Requirement already satisfied: setuptools>=36.2.1 in /var/lang/lib/python3.8/site-packages (from pipenv) (49.2.1)
Collecting virtualenv-clone>=0.2.5
Downloading virtualenv_clone-0.5.4-py2.py3-none-any.whl (6.6 kB)
Collecting virtualenv
Downloading virtualenv-20.4.3-py2.py3-none-any.whl (7.2 MB)
Requirement already satisfied: pip>=18.0 in /var/lang/lib/python3.8/site-packages (from pipenv) (21.0.1)
Collecting certifi
Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting html5lib<2.0,>=1.0
Downloading html5lib-1.1-py2.py3-none-any.whl (112 kB)
Collecting packaging<21.0,>=20.4
Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
Collecting shellingham<2.0,>=1.1
Downloading shellingham-1.4.0-py2.py3-none-any.whl (9.4 kB)
Collecting cleo<0.9.0,>=0.8.1
Downloading cleo-0.8.1-py2.py3-none-any.whl (21 kB)
Collecting crashtest<0.4.0,>=0.3.0
Downloading crashtest-0.3.1-py3-none-any.whl (7.0 kB)
Collecting tomlkit<1.0.0,>=0.7.0
Downloading tomlkit-0.7.0-py2.py3-none-any.whl (32 kB)
Collecting poetry-core<1.1.0,>=1.0.3
Downloading poetry_core-1.0.3-py2.py3-none-any.whl (424 kB)
Collecting keyring<22.0.0,>=21.2.0
Downloading keyring-21.8.0-py3-none-any.whl (32 kB)
Collecting pkginfo<2.0,>=1.4
Downloading pkginfo-1.7.0-py2.py3-none-any.whl (25 kB)
Collecting clikit<0.7.0,>=0.6.2
Downloading clikit-0.6.2-py2.py3-none-any.whl (91 kB)
Collecting requests<3.0,>=2.18
Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting requests-toolbelt<0.10.0,>=0.9.1
Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Collecting pexpect<5.0.0,>=4.7.0
Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting cachy<0.4.0,>=0.3.0
Downloading cachy-0.3.0-py2.py3-none-any.whl (20 kB)
Collecting cachecontrol[filecache]<0.13.0,>=0.12.4
Downloading CacheControl-0.12.6-py2.py3-none-any.whl (19 kB)
Collecting msgpack>=0.5.2
Downloading msgpack-1.0.2-cp38-cp38-manylinux1_x86_64.whl (302 kB)
Collecting lockfile>=0.9
Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
Collecting pylev<2.0,>=1.3
Downloading pylev-1.3.0-py2.py3-none-any.whl (4.9 kB)
Collecting pastel<0.3.0,>=0.2.0
Downloading pastel-0.2.1-py2.py3-none-any.whl (6.0 kB)
Collecting six>=1.9
Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting webencodings
Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting SecretStorage>=3.2
Downloading SecretStorage-3.3.1-py3-none-any.whl (15 kB)
Collecting jeepney>=0.4.2
Downloading jeepney-0.6.0-py3-none-any.whl (45 kB)
Collecting pyparsing>=2.0.2
Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting ptyprocess>=0.5
Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting chardet<5,>=3.0.2
Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting idna<3,>=2.5
Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3<1.27,>=1.21.1
Downloading urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
Collecting cryptography>=2.0
Downloading cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
Collecting cffi>=1.12
Downloading cffi-1.14.5-cp38-cp38-manylinux1_x86_64.whl (411 kB)
Collecting pycparser
Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting distlib<1,>=0.3.1
Downloading distlib-0.3.1-py2.py3-none-any.whl (335 kB)
Collecting filelock<4,>=3.0.0
Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting appdirs<2,>=1.4.3
Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Installing collected packages: pycparser, urllib3, idna, chardet, cffi, certifi, requests, pylev, pastel, msgpack, jeepney, cryptography, crashtest, webencodings, six, SecretStorage, pyparsing, ptyprocess, lockfile, filelock, distlib, clikit, cachecontrol, appdirs, virtualenv-clone, virtualenv, tomlkit, shellingham, requests-toolbelt, poetry-core, pkginfo, pexpect, packaging, keyring, html5lib, cleo, cachy, poetry, pipenv
Successfully installed SecretStorage-3.3.1 appdirs-1.4.4 cachecontrol-0.12.6 cachy-0.3.0 certifi-2020.12.5 cffi-1.14.5 chardet-4.0.0 cleo-0.8.1 clikit-0.6.2 crashtest-0.3.1 cryptography-3.4.7 distlib-0.3.1 filelock-3.0.12 html5lib-1.1 idna-2.10 jeepney-0.6.0 keyring-21.8.0 lockfile-0.12.2 msgpack-1.0.2 packaging-20.9 pastel-0.2.1 pexpect-4.8.0 pipenv-2020.11.15 pkginfo-1.7.0 poetry-1.1.6 poetry-core-1.0.3 ptyprocess-0.7.0 pycparser-2.20 pylev-1.3.0 pyparsing-2.4.7 requests-2.25.1 requests-toolbelt-0.9.1 shellingham-1.4.0 six-1.15.0 tomlkit-0.7.0 urllib3-1.26.4 virtualenv-20.4.3 virtualenv-clone-0.5.4 webencodings-0.5.1
Removing intermediate container fd0a1868ac36
---> 19e59c530325
Step 6/9 : WORKDIR /var/dependencies
---> Running in dbe9f58c3bde
Removing intermediate container dbe9f58c3bde
---> 595da7e52ae0
Step 7/9 : COPY Pipfile* pyproject* poetry* requirements.tx[t] ./
---> 6023d400a134
Step 8/9 : RUN [ -f 'Pipfile' ] && pipenv lock -r >requirements.txt; [ -f 'poetry.lock' ] && poetry export --with-credentials --format requirements.txt --output requirements.txt; [ -f 'requirements.txt' ] && pip install -r requirements.txt -t .;
---> Running in 1f799baa02e9
Collecting attrs==20.3.0
Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Collecting botocore==1.20.53
Downloading botocore-1.20.53-py2.py3-none-any.whl (7.4 MB)
Collecting jmespath==0.10.0
Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting jsonschema==3.2.0
Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting pynamodb==5.0.3
Downloading pynamodb-5.0.3-py3-none-any.whl (56 kB)
Collecting pyrsistent==0.17.3
Downloading pyrsistent-0.17.3.tar.gz (106 kB)
Collecting python-dateutil==2.8.1
Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting six==1.15.0
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting urllib3==1.26.4
Using cached urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
Collecting setuptools
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
setuptools from https://files.pythonhosted.org/packages/ae/42/2876a3a136f8bfa9bd703518441c8db78ff1eeaddf174baa85c083c1fd15/setuptools-56.0.0-py3-none-any.whl#sha256=7430499900e443375ba9449a9cc5d78506b801e929fef4a186496012f93683b5 (from jsonschema==3.2.0->-r requirements.txt (line 10))
The command '/bin/sh -c [ -f 'Pipfile' ] && pipenv lock -r >requirements.txt; [ -f 'poetry.lock' ] && poetry export --with-credentials --format requirements.txt --output requirements.txt; [ -f 'requirements.txt' ] && pip install -r requirements.txt -t .;' returned a non-zero code: 1
jsii.errors.JavaScriptError:
Error: docker exited with status 1
at dockerExec (/tmp/jsii-kernel-gEBMQt/node_modules/@aws-cdk/core/lib/bundling.js:207:15)
at Function.fromAsset (/tmp/jsii-kernel-gEBMQt/node_modules/@aws-cdk/core/lib/bundling.js:66:9)
at Object.bundle (/tmp/jsii-kernel-gEBMQt/node_modules/@aws-cdk/aws-lambda-python/lib/bundling.js:36:43)
at new PythonFunction (/tmp/jsii-kernel-gEBMQt/node_modules/@aws-cdk/aws-lambda-python/lib/function.js:39:30)
at /tmp/tmpq1on2qlt/lib/program.js:8154:58
at Kernel._wrapSandboxCode (/tmp/tmpq1on2qlt/lib/program.js:8582:24)
at Kernel._create (/tmp/tmpq1on2qlt/lib/program.js:8154:34)
at Kernel.create (/tmp/tmpq1on2qlt/lib/program.js:7895:29)
at KernelHost.processRequest (/tmp/tmpq1on2qlt/lib/program.js:9479:36)
at KernelHost.run (/tmp/tmpq1on2qlt/lib/program.js:9442:22)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "app.py", line 75, in <module>
main()
File "app.py", line 51, in main
APIStack(
File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_runtime.py", line 83, in __call__
inst = super().__call__(*args, **kwargs)
File "/home/victor/dev/geospatial-data-lake/infrastructure/api_stack.py", line 58, in __init__
datasets_endpoint_lambda = LambdaEndpoint(
File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_runtime.py", line 83, in __call__
inst = super().__call__(*args, **kwargs)
File "/home/victor/dev/geospatial-data-lake/infrastructure/constructs/lambda_endpoint.py", line 17, in __init__
super().__init__(
File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/aws_cdk/aws_lambda_python/__init__.py", line 252, in __init__
jsii.create(PythonFunction, self, [scope, id, props])
File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_kernel/__init__.py", line 275, in create
response = self.provider.create(
File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 344, in create
return self._process.send(request, CreateResponse)
File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 326, in send
raise JSIIError(resp.error) from JavaScriptError(resp.stack)
jsii.errors.JSIIError: docker exited with status 1
Environment
- CDK CLI Version : 1.98.0 (build 79f4512)
- Framework Version: aws-cli/1.19.52 Python/3.8.6 Linux/5.11.13-arch1-1 botocore/1.20.53
- Node.js Version: v14.15.1
- OS : Arch Linux
- Language (Version): Python (3.8.6)
Other
Looks like it’s this Poetry issue, which means one of these workarounds have to be used.
This is 🐛 Bug Report
Issue Analytics
- State:
- Created 2 years ago
- Reactions:1
- Comments:6 (4 by maintainers)
Top Results From Across the Web
Building Lambdas with Poetry - Chariot Solutions
The problem that I was trying to solve was building a Lambda deployment bundle. These are ZIP files that contain your program code,...
Read more >Using Poetry with Serverless Framework to Deploy Python ...
This guide describes how to use Poetry along with the Serverless framework to deploy AWS Lambda functions.
Read more >Python Poetry for AWS Lambda Functions - Binx
Run poetry install to generate a poetry.lock file and install all dependencies. Finally, we can test the function by running: export AWS_PROFILE ......
Read more >Serverless Framework - Python and Requirements.txt
If so what am I missing?) Is there documentation on what is required to do this and how it works? Is it serverless...
Read more >Resolve "Unable to import module" errors from Python ...
To resolve this error, create a deployment package or Lambda layer that includes the libraries that you want to use in your Python...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Just to give some light into the matter, this problem is not technically an issue with the CDK but with pip. The issue was reported, solved and reverted, so we are still experiencing this issue: https://github.com/pypa/pip/issues/9243 and https://github.com/pypa/pip/issues/9644
My workaround was to remove the poetry.lock file and manually export to a requirements.txt file using
poetry export -o requirements.txt --without-hashes
. Take into account that it includes the line--index-url
in the file and in my case it was completely omitting the PIP_INDEX_URL environment variable turning into a new problem since I was using a private repo. Removing that--index-url
line solved the issue.The only thing that could be done while pip fixes its problem, is to have the possibility of telling to the CDK that during the export we want to include the flag
--without-hashes
.That could be a workaround to solve this bug as well: https://github.com/aws/aws-cdk/issues/19232
setuptools isn’t included in the export from Poetry, because it’s excluded from poetry.lock and the Poetry section of pyproject.toml. It looks like pip installs it anyway. So it looks like the way to fix this would be for CDK to either add some version of setuptools with hashes to the requirements.txt (risky) or to force
pip
to not install setuptools (--no-deps
?).