Pipenv does not add "requires" packages in pyproject.toml to Pipfile.lock when installing from VCS like GitHub

Issue description

pipenv does not add requires packages from the pyproject.toml file when installing from GitHub. This can create build issues given that an application works locally since pipenv installed the requires packages, but when you build it in a CI/CD pipeline with pipenv install --deploy, pipenv installs from Pipfile.lock and these packages are missing. This resulted in a bug where I couldn’t figure out why my production images lacked the library dependencies from my GitHub install, because they existed locally, but I realized they were not added to the Pipfile.lock which I rely upon for production builds.

Expected result

I would expect pipenv to add the requires packages from pyproject.toml to the Pipfile.lock when installing from Github.

Actual result

pipenv installs the package and its requires packages, but does not add the requires packages to the Pipfile.lock

Steps to replicate

1. Install package from GitHub that uses a pyproject.toml

pipenv install git+https://github.com/mtzgroup/tccloud.git@develop#egg=tccloud

1. Note that the packages has dependencies. Here is the pyproject.toml

[build-system]
requires = ["flit_core >=2,<4"]
build-backend = "flit_core.buildapi"

[tool.flit.metadata]
module = "tccloud"
author = "Colton Hicks"
author-email = "[e-mail]"
home-page = "https://github.com/coltonbh/terachem-cloud-pyclient"
classifiers = [
    "Intended Audience :: Science/Research",
    "Operating System :: OS Independent",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3 :: Only",
    "Programming Language :: Python :: 3.6",
    "Programming Language :: Python :: 3.7",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Topic :: Software Development :: Libraries :: Python Modules",
    "Topic :: Software Development :: Libraries",
    "Topic :: Software Development",
    "Typing :: Typed",
    "License :: OSI Approved :: MIT License",

    
    ]
description-file = "README.md"

requires = [
    "qcelemental >= 0.17.0",
    "httpx >= 0.16.1",
    "toml >= 0.10.2",
]

requires-python = ">=3.6"

[tool.flit.metadata.requires-extra]
test = [
    "pytest >=6.2.1",
    "pytest-cov >=2.10.1,<3.0.0",
    "pytest-httpx >=0.10.1",
    "pytest-mock >=3.5.1",
    "coverage >=5.3.1,<6.0",
    "mypy ==0.790",
    "black >=20.8b1,<21.0b0",
    "isort >=5.7.0,<6.0.0"
]

dev = [
    "flake8 >=3.8.4",
    "pre-commit >= 2.9.3",
]

[tool.pytest.ini_options]
testpaths = "tests/"

[tool.black]
line-length = 88

[tool.isort]
multi_line_output = 3
include_trailing_comma = true
force_grid_wrap = 0
use_parentheses = true
line_length = 88

[tool.coverage.run]
branch = true
omit = [
    "*/tests/*",
    "*/migrations/*",
    "*site-packages*",
    "*__init__.py",
]

1. After install, note that the dependencies were not added to the Pipfile.lock

cat Pipfile.lock 
{
    "_meta": {
        "hash": {
            "sha256": "8953ef6fd1a306c1baa534710875fc31219cf08660f211811c62ab37334ecd17"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.9"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "tccloud": {
            "git": "https://github.com/mtzgroup/tccloud.git",
            "ref": "4ff980bd00a969786ba7e4295d0929c6d8ea3e51"
        }
    },
    "develop": {}
}

1. Note that the dependencies were locally installed:

pipenv graph
idna==3.1
tccloud==0.1.1
  - httpx [required: >=0.16.1, installed: 0.16.1]
    - certifi [required: Any, installed: 2020.12.5]
    - httpcore [required: ==0.12.*, installed: 0.12.3]
      - h11 [required: ==0.*, installed: 0.12.0]
      - sniffio [required: ==1.*, installed: 1.2.0]
    - rfc3986 [required: >=1.3,<2, installed: 1.4.0]
    - sniffio [required: Any, installed: 1.2.0]
  - qcelemental [required: >=0.17.0, installed: 0.18.0]
    - numpy [required: >=1.12.0, installed: 1.20.1]
    - pint [required: >=0.10.0, installed: 0.16.1]
      - packaging [required: Any, installed: 20.9]
        - pyparsing [required: >=2.0.2, installed: 2.4.7]
    - pydantic [required: >=1.5.0,!=1.6.0, installed: 1.7.3]
  - toml [required: >=0.10.2, installed: 0.10.2]

---

Please run $ pipenv --support, and paste the results here. Don’t put backticks (`) around it! The output already contains Markdown formatting.

$ pipenv --support

Pipenv version: '2020.11.15'

Pipenv location: '/usr/local/lib/python3.9/site-packages/pipenv'

Python location: '/usr/local/bin/python'

Python installations found:

• 3.9.1: /usr/local/bin/python3
• 3.9.1: /usr/local/bin/python3.9
• 3.7.3: /usr/bin/python3
• 3.7.3: /usr/bin/python3.7m
• 3.7.3: /usr/bin/python3.7
• 2.7.16: /usr/bin/python2
• 2.7.16: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.9.1',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.19.121-linuxkit',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Tue Dec 1 17:50:32 UTC 2020',
 'python_full_version': '3.9.1',
 'python_version': '3.9',
 'sys_platform': 'linux'}

System environment variables:

• HOSTNAME
• PYTHON_VERSION
• PWD
• HOME
• LANG
• GPG_KEY
• TERM
• SHLVL
• PYTHON_PIP_VERSION
• PYTHON_GET_PIP_SHA256
• PYTHON_GET_PIP_URL
• PATH
• _
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
• LANG: C.UTF-8
• PWD: /

---

[y4n9squared]

In the documentation here (https://pipenv.pypa.io/en/latest/advanced/#pipfile-vs-setup-py) it says

You can also do this:

$ pipenv install -e . This will tell Pipenv to lock all your setup.py–declared dependencies.

However, pipenv install -e . (tested with version 2020.11.15) seems to have the same behavior as noted in this issue - the setup.py-declared dependencies are not locked. Is the documentation wrong or am I misunderstanding?

[coltonbh]

Hi @dudil – Thanks for the thoughts!

Good note on the documentation; however, I disagree that this means the behavior I see means pipenv is working as designed. pipenv is supposed to be able to correctly install libraries into your application and keep a record of those installs in the Pipfile and Pipfile.lock. The failure I am seeing is a failure of pipenv to correctly document the install of a library into my application–not an issue of the same library having a Pipfile and a setup.py file and pipenv not respecting the setup.py file.

I.e., I am requesting the installation of an external library into my application, and pipenv is making the mistakes noted above–it installs the package and all its dependencies correctly, but fails to add the dependencies to the Pipfile.lock. This means that the application builds are not longer stable because future pipenv installs will use the Pipfile.lock and it will lack the required dependency packages.

So I believe this is a failure of pipenv’s intended behavior, not an issue of having differences between a setup.py file (or pyproject.toml) and the Pipfile. pipenv is not documenting the installation of an external library correctly.