(aws_lambda_python): Poetry exported requirements not readable by pip

Pip refuses to install the requirements generated by poetry export.

Reproduction Steps

1. Create a simple pyproject.toml and poetry.lock in a directory with existing Lambda job code (poetry init && poetry add jsonschema="*" pynamodb="*")
2. Instantiate a PythonFunction for the directory above
3. cdk synth

What did you expect to happen?

The build should install the packages mentioned in poetry.lock.

What actually happened?

Step 1/9 : ARG IMAGE=amazon/aws-sam-cli-build-image-python3.7
Step 2/9 : FROM $IMAGE
 ---> 8aae4ac71397
Step 3/9 : RUN yum -q list installed rsync &>/dev/null || yum install -y rsync
 ---> Using cache
 ---> 97cffc20d856
Step 4/9 : RUN pip install --upgrade pip
 ---> Running in 71bfdf9b78cb
Collecting pip
  Downloading pip-21.0.1-py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
Successfully installed pip-21.0.1
Removing intermediate container 71bfdf9b78cb
 ---> 4509190dda1a
Step 5/9 : RUN pip install pipenv poetry
 ---> Running in fd0a1868ac36
Collecting pipenv
  Downloading pipenv-2020.11.15-py2.py3-none-any.whl (3.9 MB)
Collecting poetry
  Downloading poetry-1.1.6-py2.py3-none-any.whl (172 kB)
Requirement already satisfied: setuptools>=36.2.1 in /var/lang/lib/python3.8/site-packages (from pipenv) (49.2.1)
Collecting virtualenv-clone>=0.2.5
  Downloading virtualenv_clone-0.5.4-py2.py3-none-any.whl (6.6 kB)
Collecting virtualenv
  Downloading virtualenv-20.4.3-py2.py3-none-any.whl (7.2 MB)
Requirement already satisfied: pip>=18.0 in /var/lang/lib/python3.8/site-packages (from pipenv) (21.0.1)
Collecting certifi
  Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting html5lib<2.0,>=1.0
  Downloading html5lib-1.1-py2.py3-none-any.whl (112 kB)
Collecting packaging<21.0,>=20.4
  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
Collecting shellingham<2.0,>=1.1
  Downloading shellingham-1.4.0-py2.py3-none-any.whl (9.4 kB)
Collecting cleo<0.9.0,>=0.8.1
  Downloading cleo-0.8.1-py2.py3-none-any.whl (21 kB)
Collecting crashtest<0.4.0,>=0.3.0
  Downloading crashtest-0.3.1-py3-none-any.whl (7.0 kB)
Collecting tomlkit<1.0.0,>=0.7.0
  Downloading tomlkit-0.7.0-py2.py3-none-any.whl (32 kB)
Collecting poetry-core<1.1.0,>=1.0.3
  Downloading poetry_core-1.0.3-py2.py3-none-any.whl (424 kB)
Collecting keyring<22.0.0,>=21.2.0
  Downloading keyring-21.8.0-py3-none-any.whl (32 kB)
Collecting pkginfo<2.0,>=1.4
  Downloading pkginfo-1.7.0-py2.py3-none-any.whl (25 kB)
Collecting clikit<0.7.0,>=0.6.2
  Downloading clikit-0.6.2-py2.py3-none-any.whl (91 kB)
Collecting requests<3.0,>=2.18
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting requests-toolbelt<0.10.0,>=0.9.1
  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Collecting pexpect<5.0.0,>=4.7.0
  Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting cachy<0.4.0,>=0.3.0
  Downloading cachy-0.3.0-py2.py3-none-any.whl (20 kB)
Collecting cachecontrol[filecache]<0.13.0,>=0.12.4
  Downloading CacheControl-0.12.6-py2.py3-none-any.whl (19 kB)
Collecting msgpack>=0.5.2
  Downloading msgpack-1.0.2-cp38-cp38-manylinux1_x86_64.whl (302 kB)
Collecting lockfile>=0.9
  Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
Collecting pylev<2.0,>=1.3
  Downloading pylev-1.3.0-py2.py3-none-any.whl (4.9 kB)
Collecting pastel<0.3.0,>=0.2.0
  Downloading pastel-0.2.1-py2.py3-none-any.whl (6.0 kB)
Collecting six>=1.9
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting webencodings
  Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting SecretStorage>=3.2
  Downloading SecretStorage-3.3.1-py3-none-any.whl (15 kB)
Collecting jeepney>=0.4.2
  Downloading jeepney-0.6.0-py3-none-any.whl (45 kB)
Collecting pyparsing>=2.0.2
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting ptyprocess>=0.5
  Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
Collecting cryptography>=2.0
  Downloading cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
Collecting cffi>=1.12
  Downloading cffi-1.14.5-cp38-cp38-manylinux1_x86_64.whl (411 kB)
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting distlib<1,>=0.3.1
  Downloading distlib-0.3.1-py2.py3-none-any.whl (335 kB)
Collecting filelock<4,>=3.0.0
  Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting appdirs<2,>=1.4.3
  Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Installing collected packages: pycparser, urllib3, idna, chardet, cffi, certifi, requests, pylev, pastel, msgpack, jeepney, cryptography, crashtest, webencodings, six, SecretStorage, pyparsing, ptyprocess, lockfile, filelock, distlib, clikit, cachecontrol, appdirs, virtualenv-clone, virtualenv, tomlkit, shellingham, requests-toolbelt, poetry-core, pkginfo, pexpect, packaging, keyring, html5lib, cleo, cachy, poetry, pipenv
Successfully installed SecretStorage-3.3.1 appdirs-1.4.4 cachecontrol-0.12.6 cachy-0.3.0 certifi-2020.12.5 cffi-1.14.5 chardet-4.0.0 cleo-0.8.1 clikit-0.6.2 crashtest-0.3.1 cryptography-3.4.7 distlib-0.3.1 filelock-3.0.12 html5lib-1.1 idna-2.10 jeepney-0.6.0 keyring-21.8.0 lockfile-0.12.2 msgpack-1.0.2 packaging-20.9 pastel-0.2.1 pexpect-4.8.0 pipenv-2020.11.15 pkginfo-1.7.0 poetry-1.1.6 poetry-core-1.0.3 ptyprocess-0.7.0 pycparser-2.20 pylev-1.3.0 pyparsing-2.4.7 requests-2.25.1 requests-toolbelt-0.9.1 shellingham-1.4.0 six-1.15.0 tomlkit-0.7.0 urllib3-1.26.4 virtualenv-20.4.3 virtualenv-clone-0.5.4 webencodings-0.5.1
Removing intermediate container fd0a1868ac36
 ---> 19e59c530325
Step 6/9 : WORKDIR /var/dependencies
 ---> Running in dbe9f58c3bde
Removing intermediate container dbe9f58c3bde
 ---> 595da7e52ae0
Step 7/9 : COPY Pipfile* pyproject* poetry* requirements.tx[t] ./
 ---> 6023d400a134
Step 8/9 : RUN [ -f 'Pipfile' ] && pipenv lock -r >requirements.txt;     [ -f 'poetry.lock' ] && poetry export --with-credentials --format requirements.txt --output requirements.txt;     [ -f 'requirements.txt' ] && pip install -r requirements.txt -t .;
 ---> Running in 1f799baa02e9
Collecting attrs==20.3.0
  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Collecting botocore==1.20.53
  Downloading botocore-1.20.53-py2.py3-none-any.whl (7.4 MB)
Collecting jmespath==0.10.0
  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting jsonschema==3.2.0
  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Collecting pynamodb==5.0.3
  Downloading pynamodb-5.0.3-py3-none-any.whl (56 kB)
Collecting pyrsistent==0.17.3
  Downloading pyrsistent-0.17.3.tar.gz (106 kB)
Collecting python-dateutil==2.8.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting six==1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting urllib3==1.26.4
  Using cached urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
Collecting setuptools
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    setuptools from https://files.pythonhosted.org/packages/ae/42/2876a3a136f8bfa9bd703518441c8db78ff1eeaddf174baa85c083c1fd15/setuptools-56.0.0-py3-none-any.whl#sha256=7430499900e443375ba9449a9cc5d78506b801e929fef4a186496012f93683b5 (from jsonschema==3.2.0->-r requirements.txt (line 10))
The command '/bin/sh -c [ -f 'Pipfile' ] && pipenv lock -r >requirements.txt;     [ -f 'poetry.lock' ] && poetry export --with-credentials --format requirements.txt --output requirements.txt;     [ -f 'requirements.txt' ] && pip install -r requirements.txt -t .;' returned a non-zero code: 1
jsii.errors.JavaScriptError: 
  Error: docker exited with status 1
      at dockerExec (/tmp/jsii-kernel-gEBMQt/node_modules/@aws-cdk/core/lib/bundling.js:207:15)
      at Function.fromAsset (/tmp/jsii-kernel-gEBMQt/node_modules/@aws-cdk/core/lib/bundling.js:66:9)
      at Object.bundle (/tmp/jsii-kernel-gEBMQt/node_modules/@aws-cdk/aws-lambda-python/lib/bundling.js:36:43)
      at new PythonFunction (/tmp/jsii-kernel-gEBMQt/node_modules/@aws-cdk/aws-lambda-python/lib/function.js:39:30)
      at /tmp/tmpq1on2qlt/lib/program.js:8154:58
      at Kernel._wrapSandboxCode (/tmp/tmpq1on2qlt/lib/program.js:8582:24)
      at Kernel._create (/tmp/tmpq1on2qlt/lib/program.js:8154:34)
      at Kernel.create (/tmp/tmpq1on2qlt/lib/program.js:7895:29)
      at KernelHost.processRequest (/tmp/tmpq1on2qlt/lib/program.js:9479:36)
      at KernelHost.run (/tmp/tmpq1on2qlt/lib/program.js:9442:22)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "app.py", line 75, in <module>
    main()
  File "app.py", line 51, in main
    APIStack(
  File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_runtime.py", line 83, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/home/victor/dev/geospatial-data-lake/infrastructure/api_stack.py", line 58, in __init__
    datasets_endpoint_lambda = LambdaEndpoint(
  File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_runtime.py", line 83, in __call__
    inst = super().__call__(*args, **kwargs)
  File "/home/victor/dev/geospatial-data-lake/infrastructure/constructs/lambda_endpoint.py", line 17, in __init__
    super().__init__(
  File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/aws_cdk/aws_lambda_python/__init__.py", line 252, in __init__
    jsii.create(PythonFunction, self, [scope, id, props])
  File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_kernel/__init__.py", line 275, in create
    response = self.provider.create(
  File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 344, in create
    return self._process.send(request, CreateResponse)
  File "/home/victor/dev/geospatial-data-lake/.venv/lib/python3.8/site-packages/jsii/_kernel/providers/process.py", line 326, in send
    raise JSIIError(resp.error) from JavaScriptError(resp.stack)
jsii.errors.JSIIError: docker exited with status 1

Environment

• CDK CLI Version : 1.98.0 (build 79f4512)
• Framework Version: aws-cli/1.19.52 Python/3.8.6 Linux/5.11.13-arch1-1 botocore/1.20.53
• Node.js Version: v14.15.1
• OS : Arch Linux
• Language (Version): Python (3.8.6)

Other

Looks like it’s this Poetry issue, which means one of these workarounds have to be used.

---

This is 🐛 Bug Report

[AbendGithub]

Just to give some light into the matter, this problem is not technically an issue with the CDK but with pip. The issue was reported, solved and reverted, so we are still experiencing this issue: https://github.com/pypa/pip/issues/9243 and https://github.com/pypa/pip/issues/9644

My workaround was to remove the poetry.lock file and manually export to a requirements.txt file using poetry export -o requirements.txt --without-hashes. Take into account that it includes the line --index-url in the file and in my case it was completely omitting the PIP_INDEX_URL environment variable turning into a new problem since I was using a private repo. Removing that --index-url line solved the issue.

The only thing that could be done while pip fixes its problem, is to have the possibility of telling to the CDK that during the export we want to include the flag --without-hashes.

That could be a workaround to solve this bug as well: https://github.com/aws/aws-cdk/issues/19232

[l0b0]

setuptools isn’t included in the export from Poetry, because it’s excluded from poetry.lock and the Poetry section of pyproject.toml. It looks like pip installs it anyway. So it looks like the way to fix this would be for CDK to either add some version of setuptools with hashes to the requirements.txt (risky) or to force pip to not install setuptools (--no-deps?).