Errors when locking after system installs in Docker

I’ve been running into what seemed to be an intermittent issue for several weeks now and have finally been able to narrow it down to a repeatable test case.

When trying to install a new package system wide, with no virtualenv for the root user, and specific package present in the Pipfile, the install process will complete and the lock process will fail. pylint, pylint-django, nose-watch and arrow are some of the packages in the my Pipfile that seem to be involved.

Versions:

I’m running in Docker specifically, but I suspect a fresh Linux install would probably result in the same. Python 2.7 and 3.6 pipenv - 9.0.3 pip - 9.0.1 pew - 1.1.2 virtualenv - 15.1.0

Steps:

1. In a Docker container (Dockerfile below)(docker run -ti -u root <<image>> bash)
2. run pipenv install --system django (but it’s the same with any package)
3. Results in

   Installing six…
   Requirement already satisfied: six in /usr/local/lib/python2.7/site-packages
   
   Adding six to Pipfile's [packages]…
   Locking [dev-packages] dependencies…
   CRITICAL:pip.utils:Error [Errno 2] No such file or directory while executing command python setup.py egg_info
   <<< rest of traceback below >>>
   

4. run any command that creates a virtualenv, such as pipenv run ls
5. run pipenv install --system six
6. This completes successfully,

   Installing six…
   Requirement already satisfied: six in /usr/local/lib/python2.7/site-packages
   
   Adding six to Pipfile's [packages]…
   Locking [dev-packages] dependencies…
   Locking [packages] dependencies…
   Updated Pipfile.lock (0a3864)!
   

7. rm -r /root/.virtualenvs/app-MuJj66st/
8. now pipenv install --system six reverts to previous error

Dockerfile

FROM python:2.7
RUN pip install pipenv
RUN mkdir /srv/app
WORKDIR /srv/app
COPY Pipfile* ./
RUN pipenv install --deploy --system

Pipfile

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[dev-packages]
pylint = "*"


[packages]
six = '*'

Full Traceback

Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 1934, in install
    do_lock(system=system, pre=pre)
  File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 1041, in do_lock
    pre=pre
  File "/usr/local/lib/python2.7/site-packages/pipenv/utils.py", line 545, in resolve_deps
    resolved_tree = actually_resolve_reps(deps, index_lookup, markers_lookup, project, sources, verbose, clear, pre)
  File "/usr/local/lib/python2.7/site-packages/pipenv/utils.py", line 507, in actually_resolve_reps
    resolved_tree.update(resolver.resolve(max_rounds=PIPENV_MAX_ROUNDS))
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/piptools/resolver.py", line 200, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/piptools/resolver.py", line 278, in _iter_dependencies
    for dependency in self.repository.get_dependencies(ireq):
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 153, in get_dependencies
    result = reqset._prepare_file(self.finder, ireq)
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/pip/req/req_set.py", line 639, in _prepare_file
    abstract_dist.prep_for_dist()
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/pip/req/req_set.py", line 134, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/pip/req/req_install.py", line 438, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/usr/local/lib/python2.7/site-packages/pipenv/patched/pip/utils/__init__.py", line 667, in call_subprocess
    cwd=cwd, env=env)
  File "/usr/local/lib/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/usr/local/lib/python2.7/subprocess.py", line 1025, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Workaround

For know I’m running pipenv install --system --nolock <<pkg>>, then pipenv lock to get the package installed and successfully regenerate the lockfile.

[jsatt]

@kennethreitz, @techalchemy, I think in this day, using Docker for development is becoming quite common and the very nature of Docker encourages system level installation as opposed to virtualenvs, as indicated in your own documentation. In such a development environment, how is the developer intended to add a new package to the Pipfile, install that package at the system level with all other installed packages and update the lockfile for committing later? To completely restrict the developer, as commit 52ea1d4 does, requires the user to do each of these steps separately and manually, quickly chipping away at pipenv’s usefulness to the project at all.

If there is another workflow you intend developers to use in this sort of environment I think it’d be great to have it documented somewhere, but if not, I think we should try to figure out a workflow that works better in what is likely to be a more common environment as pipenv continues to gain adoption.

[jtratner]

@kennethreitz - if I put up a proposed “standard Docker workflow” are you cool with adding it to the docs? (Any suggestions on what that would be, heh?)

From my perspective, I actually came to pipenv because I was eager to use it as a sane way to manage requirements.txt in Docker and conda environments - I think there’s a lot to offer there (even if you’re installing a couple particularly difficult to manage scientific packages with conda) On Fri, Feb 23, 2018 at 7:30 AM Kenneth Reitz [e-mail] wrote:

Pipenv and pip are very friendly tools to use together.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pypa/pipenv/issues/1428#issuecomment-368041413, or mute the thread https://github.com/notifications/unsubscribe-auth/ABhjq_IwM2WOFAQdVSlwMYpWnniQX6YOks5tXtmTgaJpZM4SGJD9 .