Packages in dev-packages cannot set editable flag if overriding from packages

Issue description

I expect to be able to override elements in [packages] by reproducing them in [dev-packages] to declare development-time differences.

For example, the following pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
funniest = {editable = true,path = "."}

[packages]
funniest = {path = "."}

[requires]
python_version = "3.7"

with setup.py:

  from setuptools import setup

  setup(name='funniest',
        version='0.1',
        description='The funniest joke in the world',
        url='http://github.com/storborg/funniest',
        author='Flying Circus',
        author_email='[e-mail]',
        license='MIT',
        packages=['funniest'],
        zip_safe=False)

(example taken from the python packaging docs)

and two empty directories, funniest and foo. The reason for the foo directory is so we can be in a working directory where funniest is not importable from the cwd, as this masks the error.

Expected result

I expect that if I run pipenv install the funniest package will be installed into the virtual environment. Changes to the current working directory are not applied until reinstallation.

If I run pipenv install --dev the funniest package will be installed as editable, so the current working directory is.

That is (paths and output simplified):

example$ cd foo
example/foo$ pipenv --rm
...
example/foo$ pipenv install
...
example/foo$ pipenv run python -c 'import pkg_resources; print(pkg_resources.require("funniest")[0].location)'
~/.local/share/virtualenvs/example-BaUoFXTC/lib/python3.7/site-packages

and

example/foo$ pipenv --rm
...
example/foo$ pipenv install --dev
...
example/foo$ pipenv run python -c 'import pkg_resources; print(pkg_resources.require("funniest")[0].location)'
example/

The generated lockfile looks like:

Actual result

The editable option in [dev-packages] is masked by the version in [packages].

example$ cd foo
example/foo$ pipenv --rm
...
example/foo$ pipenv install
...
example/foo$ pipenv run python -c 'import pkg_resources; print(pkg_resources.require("funniest")[0].location)'
~/.local/share/virtualenvs/example-BaUoFXTC/lib/python3.7/site-packages

and

example/foo$ pipenv --rm
...
example/foo$ pipenv install --dev
...
example/foo$ pipenv run python -c 'import pkg_resources; print(pkg_resources.require("funniest")[0].location)'
~/.local/share/virtualenvs/example-BaUoFXTC/lib/python3.7/site-packages

The generated lockfile is:

{
    "_meta": {
        "hash": {
            "sha256": "35c958102b69fd89f6d04c31de2f9fb49cef92dbcaec95c68f0fd314274fa3c9"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "funniest": {
            "path": "."
        }
    },
    "develop": {
        "funniest": {
            "path": "."
        }
    }
}

Note the lack of editable lines in the develop entry.

Steps to replicate

Install a local package in both --dev and normal modes, differing by their editable flag. The difference is dropped during the locking stage.

---

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: '/home/matthewwilkes/.local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/bin/python3'

Python installations found:

• 3.7.3: /usr/bin/python3.7
• 3.7.3: /usr/bin/python3.7m
• 3.6.7: /usr/bin/python3.6
• 3.6.7: /usr/bin/python3.6m
• 3.5.6: /usr/bin/python3.5m
• 3.5.6: /usr/bin/python3.5
• 3.5.3: /snap/bin/pypy3
• 3.4.9: /usr/bin/python3.4
• 3.4.9: /usr/bin/python3.4m
• 2.7.15rc1: /usr/bin/python2
• 2.7.13: /usr/bin/pypy

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.7',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.15.0-46-generic',
 'platform_system': 'Linux',
 'platform_version': '#49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019',
 'python_full_version': '3.6.7',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• SESSION_MANAGER
• XDG_CONFIG_DIRS
• QT_ACCESSIBILITY
• LANG
• DISPLAY
• SHLVL
• LOGNAME
• LANGUAGE
• MANDATORY_PATH
• XDG_VTNR
• USER
• J2REDIR
• J2SDKDIR
• XAUTHORITY
• LG_WEBOS_TV_SDK_HOME
• GTK_IM_MODULE
• COLORTERM
• XDG_SESSION_ID
• JAVA_HOME
• GNOME_TERMINAL_SCREEN
• IM_CONFIG_PHASE
• XDG_SESSION_DESKTOP
• GDMSESSION
• TEXTDOMAINDIR
• DEFAULTS_PATH
• WINDOWPATH
• USERNAME
• TEXTDOMAIN
• GTK2_MODULES
• DERBY_HOME
• PWD
• DBUS_SESSION_BUS_ADDRESS
• GNOME_DESKTOP_SESSION_ID
• VTE_VERSION
• DESKTOP_SESSION
• GJS_DEBUG_TOPICS
• QT4_IM_MODULE
• CLUTTER_IM_MODULE
• GNOME_TERMINAL_SERVICE
• XDG_MENU_PREFIX
• XDG_DATA_DIRS
• QT_IM_MODULE
• MOZ_USE_XINPUT2
• S_COLORS
• XDG_SESSION_TYPE
• GJS_DEBUG_OUTPUT
• TERM
• SHELL
• GTK_MODULES
• SSH_AGENT_PID
• GNOME_SHELL_SESSION_MODE
• WEBOS_CLI_TV
• PATH
• XDG_CURRENT_DESKTOP
• XMODIFIERS
• HOME
• XDG_RUNTIME_DIR
• SSH_AUTH_SOCK
• XDG_SEAT
• GPG_AGENT_INFO
• OLDPWD
• LS_COLORS
• _
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/matthewwilkes/bin:/home/matthewwilkes/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin:/home/matthewwilkes/.local/bin
• SHELL: /usr/bin/zsh
• LANG: en_GB.UTF-8
• PWD: /secure/coding/pipenvtest

---

Contents of Pipfile (‘/secure/coding/pipenvtest/Pipfile’):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
funniest = {editable = true,path = "."}

[packages]
funniest = {path = "."}

[requires]
python_version = "3.7"

Contents of Pipfile.lock (‘/secure/coding/pipenvtest/Pipfile.lock’):

{
    "_meta": {
        "hash": {
            "sha256": "35c958102b69fd89f6d04c31de2f9fb49cef92dbcaec95c68f0fd314274fa3c9"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "funniest": {
            "path": "."
        }
    },
    "develop": {
        "funniest": {
            "path": "."
        }
    }
}

[brucedjones]

Whilst the reasoning above is solid (dev packages should never override prod packages), I agree with @sc250024 that there is a vital workflow that is going unsupported here. The equivalent in node would be npm link.

@techalchemy is there something we can do to support this? IMO i think it would work very cleanly if pipenv install --dev gave preference to packages under [dev-packages] when a collision occurs. That way it would be very clear how the collision is resolved, where right now its an implicit preference to [packages] at the expense of what is (for me at least) a super common workflow.

[sc250024]

This seems to be a valid use case that @MatthewWilkes brought up, no? I’m having a similar issue. The reason why I want editable in development is so that I can work locally, but when I’m building my project as a Docker image for distribution, I want it to be a package to make things easier for multi-stage Docker builds.

Currently this does not work if I have my-app = {editable = true,path = "."} in [packages]:

FROM python:3.8-alpine AS base

ENV PYROOT=/pyroot

ENV \
    PATH="${PYROOT}/bin:${PATH}" \
    PYTHONUSERBASE="${PYROOT}"

FROM base AS builder

WORKDIR /build

RUN pip3 install pipenv

COPY Pipfile* ./
COPY my_app my_app/
COPY setup.py .

RUN PIP_USER=1 PIP_IGNORE_INSTALLED=1 PIP_PROCESS_DEPENDENCY_LINKS=1 \
    pipenv install \
      --deploy \
      --ignore-pipfile \
      --system

FROM base

WORKDIR /app

COPY --from=builder $PYROOT/ $PYROOT/

ENTRYPOINT ["my-app"]

CMD ["--help"]

And if the container runs, I get the following error:

pkg_resources.DistributionNotFound: The 'my-app' distribution was not found and is required by the application

The reason is because the my-app egg gets placed into the /build directory in the first container because it’s editable, and doesn’t get placed in PYROOT like all of the other packages do.

What’s the recommended solution around this then?