Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Pipenv ignores custom index on the command line... or does it?

See original GitHub issue

I’m fairly new to this, so I’m not sure if I’m using pipenv right. I encountered several issues with pipenv.

I’m trying to install a package (that i’ve just made myself) from test.pypi.org. Basically, I want to run pip install my-package -i https://test.pypi.org/simple, which works fine, but in pipenv.

Problem 0: this doesn’t seem to be possible…ish:

% pipenv install my-package -i https://test.pypi.org/simple
Creating a virtualenv for this project...
Pipfile: /home/oakkitten/test7/Pipfile
Using /usr/bin/python (2.7.13) to create virtualenv...
⠋ Creating virtual environment...Already using interpreter /usr/bin/python
New python executable in /home/oakkitten/.local/share/virtualenvs/test7-wnMdLOpT/bin/python
Installing setuptools, pip, wheel...
done.

✔ Successfully created virtual environment! 
Virtualenv location: /home/oakkitten/.local/share/virtualenvs/test7-wnMdLOpT
Creating a Pipfile for this project...
Installing my-package...
⠸ Installing...
Adding my-package to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
✘ Locking Failed! 
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python2.7/dist-packages/pipenv/resolver.py", line 69, in resolve
[pipenv.exceptions.ResolutionFailure]:       req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]:       req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]:       resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 395, in resolve
[pipenv.exceptions.ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]:       ResolutionFailure: ERROR: ERROR: Could not find a version that matches my-package
[pipenv.exceptions.ResolutionFailure]:       No versions found
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches my-package
No versions found
Was https://pypi.org/simple reachable?
[pipenv.exceptions.ResolutionFailure]:       req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]:       req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]:       resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/lib/python2.7/dist-packages/pipenv/utils.py", line 395, in resolve
[pipenv.exceptions.ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]:       ResolutionFailure: ERROR: ERROR: Could not find a version that matches my-package
[pipenv.exceptions.ResolutionFailure]:       No versions found
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches my-package
No versions found
Was https://pypi.org/simple reachable?

Problem 1: Judging by the last line, it looks like pipenv only checked out https://pypi.org/simple and didn’t look at https://test.pypi.org/, right? Except python -c "import mymodule" works fine, and it is listed by pip freeze.

Looking at what pipenv launches with --verbose when i run it without -i, I see

pip install --verbose --upgrade my-package -i https://pypi.org/simple

But running it with -i https://test.pypi.org/simple results in an extra source added (even with no Pipfile):

pip install --verbose --upgrade my-package -i https://test.pypi.org/simple --extra-index-url https://pypi.org/simple

And running it with -i https://f.org --extra-index-url https://b.org --verbose results in no extra url added (although the documentation says there can be several?):

pip install --verbose --upgrade my-package -i https://f.org --extra-index-url https://b.org

During all of the mentioned calls, pipenv says:

Using pip: -i https://pypi.org/simple

This must be a mistake, right?

Problem 2: even though pipenv returns an error, it puts my-package = {index = "https://test.pypi.org/simple",version = "*"} into Pipfile. Likely because of that, all subsequent operations fail as well (such as pipenv clean). Related issue: #3808

I tried editing the Pipfile as follows:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[[source]]
name = "test"
url = "https://test.pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]

[requires]
python_version = "2.7"

Running pipenv install my-package -i test then worked without issues, with my-package = {index = "test",version = "*"}. Obviously, due to problem 1, pipenv install my-package -i pypi worked just as well, as did pipenv install my-package -ipypi (note the absence of the space). Manually adding my-package = {index = "wtf",version = "*"} and syncing worked as well. So, the

Problem 3 is that in practice index in Pipfile has no effect. This seems to be already covered in #2730, #2159 and other issues; just mentioning here for the sake of completeness.

So what are -i an --extra-index-url options, anyway? According to the documentation, these should be url(s):

--extra-index-url <extra_index_url>

    URLs to the extra PyPI compatible indexes to query for package lookups.

-i, --index <index>

    Target PyPI-compatible package index url.

But the latter makes it to the index = "test" parameter, which is not an url?..

All of this doesn’t make any sense to me, please advise.

Also, I’d like to mention these regarding the first error message:

Problem 4: the error message is duplicated (for some reason line with “69” is not). Also note the “ERROR: ERROR:” thing.

Problem 5: the error message is cryptic, including traceback for no apparent reason. it’s even worse when i tried to do pipenv install lxml, while the latter had an mismatching glibc version; the output looked like this:

An error occurred while installing lxml==4.4.0 --hash=sha256:06e5599b9c54f797a3c0f384c67705a0d621031007aa2400a6c7d17300fdb995 --ha
(many lines omitted)
7fdd9f39892ba05 --hash=sha256:f840dddded8b046edc774c88ed8d2442cdb231a68894c42c74e3a809450fae76! Will try again.
Installing initially failed dependencies...
(many lines omitted)
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: ['Collecting lxml==4.4.0 (from -r /tmp/pipenv-4i3Q8I-requirements/pipenv-jmkuYm-requirement.txt
(line 1))', '  Using cached https://files.pythonhosted.org/packages/e1/f5/5eb3b491958dcfdcfa5daae3c655ab59276bc216ca015e44743c9c22
0e9e/lxml-4.4.0.tar.gz', 'Building wheels for collected packages: lxml', '  Building wheel for lxml (setup.py): started', "  Build
ing wheel for lxml (setup.py): finished with status 'error'", '  Running setup.py clean for lxml', 'Failed to build lxml', 'Insta$
ling collected packages: lxml', '  Running setup.py install for lxml: started', "    Running setup.py install for lxml: finished w
ith status 'error'"]
(many lines omitted)
h:24:0,', '                     from src/lxml/etree.c:723:', '    /usr/include/libxslt/xsltlocale.h:20:21: fatal error: xlocale.h:
 No such file or directory', '     #include <xlocale.h>', '                         ^', '    compilation terminated.', "    Compil
e failed: command 'gcc' failed with exit status 1", '    cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInituub0
Dh.c -o tmp/xmlXPathInituub0Dh.o', '    /tmp/xmlXPathInituub0Dh.c:2:1: warning: return type defaults to \xe2\x80\x98int\xe2\x80\x9
9 [-Wimplicit-int]', '     main (int argc, char **argv) {', '     ^~~~', '    cc tmp/xmlXPathInituub0Dh.o -lxml2 -o a.out', "    e
rror: command 'gcc' failed with exit status 1", '    ----------------------------------------', 'ERROR: Command errored out with e
xit status 1: /home/bot/.local/share/virtualenvs/boot2-JoRJ2-zy/bin/python -u -c \'import sys, setuptools, tokenize; sys.argv[0] =
 \'"\'"\'/tmp/pip-install-_LZNij/lxml/setup.py\'"\'"\'; __file__=\'"\'"\'/tmp/pip-install-_LZNij/lxml/setup.py\'"\'"\';f=getattr(t
okenize, \'"\'"\'open\'"\'"\', open)(__file__);code=f.read().replace(\'"\'"\'\\r\\n\'"\'"\', \'"\'"\'\\n\'"\'"\');f.close();exec(c
ompile(code, __file__, \'"\'"\'exec\'"\'"\'))\' install --record /tmp/pip-record-XRL7Jv/install-record.txt --single-version-extern
ally-managed --compile --install-headers /home/bot/.local/share/virtualenvs/boot2-JoRJ2-zy/include/site/python2.7/lxml Check the l
ogs for full command output.']

As you can see, the compiler output is quite impossible to read.

Not including the output of $ pipenv --support as I tested this on several devices with clean installations of everything