Poetry export does not include setuptools even when it is a direct dependency.

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Fedora 31
• Poetry version: 1.0.0b4 (at the time of writing b5 is not on PyPi, will retry later on)
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/apollo13/1d0c72978531d3a5688f1e3847fa6fe6

Issue

Using the pyproject.toml file from the Gist and executing poetry lock I get the following lockfile:

[[package]]
category = "main"
description = "WSGI HTTP Server for UNIX"
name = "gunicorn"
optional = false
python-versions = ">=3.4"
version = "20.0.0"

[package.dependencies]
setuptools = ">=3.0"

[package.extras]
eventlet = ["eventlet (>=0.9.7)"]
gevent = ["gevent (>=0.13)"]
setproctitle = ["setproctitle"]
tornado = ["tornado (>=0.2)"]

[metadata]
content-hash = "20c8a1cc7d149b3ad58ca5063dfd37654f03ba65ee8b484d2faefe7ce28b7daf"
python-versions = "^3.7"

[metadata.files]
gunicorn = [
    {file = "gunicorn-20.0.0-py2.py3-none-any.whl", hash = "sha256:0806b5e8a2eb8ba9ac1be65d7b743ec896fc25f5d6cb16c5e051540157b315bb"},
    {file = "gunicorn-20.0.0.tar.gz", hash = "sha256:ef69dea4814df95e64e3f40b47b7ffedc6911c5009233be9d01cfd0d14aa3f50"},
]

setuptools is correctly discovered as dependency of gunicorn (see https://github.com/benoitc/gunicorn/blob/94ab2091173c6037b504f94e56f4e88816d540bf/setup.py#L71-L77 – it does indeed require it), but the lockfile does not contain any hashes for setuptools. If I now export to requirements.txt the file looks like this:

gunicorn==20.0.0 \
    --hash=sha256:0806b5e8a2eb8ba9ac1be65d7b743ec896fc25f5d6cb16c5e051540157b315bb \
    --hash=sha256:ef69dea4814df95e64e3f40b47b7ffedc6911c5009233be9d01cfd0d14aa3f50

Which is not installable with pip install --require-hashes --force-reinstall -r requirements.txt (note the --require-hases there):

Collecting gunicorn==20.0.0 (from -r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/60/0d/3dbda0324f5bf007f3274e5ea09f0f3bcbf0ca01a75b80ff4f1ff9f8ecfd/gunicorn-20.0.0-py2.py3-none-any.whl
Collecting setuptools>=3.0 (from gunicorn==20.0.0->-r requirements.txt (line 1))
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    setuptools>=3.0 from https://files.pythonhosted.org/packages/d9/de/554b6310ac87c5b921bc45634b07b11394fe63bc4cb5176f5240addf18ab/setuptools-41.6.0-py2.py3-none-any.whl#sha256=3e8e8505e563631e7cb110d9ad82d135ee866b8146d5efe06e42be07a72db20a (from gunicorn==20.0.0->-r requirements.txt (line 1))
WARNING: You are using pip version 19.1.1, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

If I instead add Django via poetry add Django, the poetry.lock file will contain Django’s dependencies (pytz and sqlparse):

pytz = [
    {file = "pytz-2019.3-py2.py3-none-any.whl", hash = "sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d"},
    {file = "pytz-2019.3.tar.gz", hash = "sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be"},
]
sqlparse = [
    {file = "sqlparse-0.3.0-py2.py3-none-any.whl", hash = "sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177"},
    {file = "sqlparse-0.3.0.tar.gz", hash = "sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873"},
]

and also export them to the requirements file:

django==2.2.7 \
    --hash=sha256:89c2007ca4fa5b351a51a279eccff298520783b713bf28efb89dfb81c80ea49b \
    --hash=sha256:16040e1288c6c9f68c6da2fe75ebde83c0a158f6f5d54f4c5177b0c1478c5b86
gunicorn==20.0.0 \
    --hash=sha256:0806b5e8a2eb8ba9ac1be65d7b743ec896fc25f5d6cb16c5e051540157b315bb \
    --hash=sha256:ef69dea4814df95e64e3f40b47b7ffedc6911c5009233be9d01cfd0d14aa3f50
pytz==2019.3 \
    --hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \
    --hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be
sqlparse==0.3.0 \
    --hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \
    --hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873

I fear there is some special casing going on for setuptools, it would be great if setuptools also ended up in the lock and requirement files if it is a direct dependency of on of the added packages.

[insysion]

It seems there’s not too much reasoning behind this (https://github.com/pypa/pip/issues/6459)

At the very least, we should introduce an --allow-unsafe or similar flag, if not remove the unsafe packages hard-coding altogether.

[insysion]

I am considering opening a PR for this. I plan to change the resolving behaviour so that the UNSAFE_PACKAGES are considered in the same way as any others but block them from being installed/exported unless an --allow-unsafe flag is set.

This will be my first poetry contribution so any advice, e.g. wrt anyone worth running this by before I start work, would be much appreciated.