tensorflow/html5lib "Your dependencies could not be resolved" but can't see problem

$ pipenv install --dev
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches html5lib!=1.0b1,!=1.0b2,!=1.0b3,!=1.0b4,!=1.0b5,!=1.0b6,!=1.0b7,!=1.0b8,==0.9999999,>=0.99999999pre
Tried: 1.0-reupload, 0.2, 0.9, 0.10, 0.11, 0.11.1, 0.90, 0.95, 0.99, 0.999, 0.9999, 0.99999, 0.999999, 0.9999999, 0.99999999, 0.999999999, 0.999999999, 1.0b1, 1.0b2, 1.0b3, 1.0b5, 1.0b6, 1.0b7, 1.0b8, 1.0b9, 1.0b10, 1.0b10, 1.0.1, 1.0.1

I did what it suggested: pipenv install --skip-lock and then pipenv graph

The graph shows that the only packages which ask for html5lib are dependencies of tensorflow, and they don’t seem to conflict with each other or the installed version:

tensorflow==1.3.0
  - numpy [required: >=1.11.0, installed: 1.14.1]
  - protobuf [required: >=3.3.0, installed: 3.5.2]
    - setuptools [required: Any, installed: 38.5.1]
    - six [required: >=1.9, installed: 1.11.0]
  - six [required: >=1.10.0, installed: 1.11.0]
  - tensorflow-tensorboard [required: >=0.1.0,<0.2.0, installed: 0.1.8]
    - bleach [required: ==1.5.0, installed: 1.5.0]
      - html5lib [required: >=0.999,<0.99999999,!=0.9999,!=0.99999, installed: 0.9999999]
        - six [required: Any, installed: 1.11.0]
      - six [required: Any, installed: 1.11.0]
    - html5lib [required: ==0.9999999, installed: 0.9999999]
      - six [required: Any, installed: 1.11.0]
    - markdown [required: >=2.6.8, installed: 2.6.11]
    - numpy [required: >=1.11.0, installed: 1.14.1]
    - protobuf [required: >=3.2.0, installed: 3.5.2]
      - setuptools [required: Any, installed: 38.5.1]
      - six [required: >=1.9, installed: 1.11.0]
    - six [required: >=1.10.0, installed: 1.11.0]
    - werkzeug [required: >=0.11.10, installed: 0.14.1]
    - wheel [required: >=0.26, installed: 0.30.0]
  - wheel [required: >=0.26, installed: 0.30.0]

So we have:

html5lib [required: >=0.999,<0.99999999,!=0.9999,!=0.99999, installed: 0.9999999]

and

html5lib [required: ==0.9999999, installed: 0.9999999]

That package uses a pretty horrible versioning scheme, but the required versions don’t seem to conflict.

One requires 0.9999999 (seven zeros, what we have installed) and the other requires <0.99999999 (eight zeros). Should be ok?

[kennethreitz]

omg is this real

[anentropic]

Well I found the problem and a solution, if not an explanation for how it happened.

I spent a while stepping through the underlying code, i.e.

from pipenv.patched.piptools.repositories import PyPIRepository
from pipenv.patched.piptools.scripts.compile import get_pip_command
from pipenv.patched.notpip.req.req_install import InstallRequirement
from pipenv.patched.piptools.resolver import Resolver

pip_command = get_pip_command()
pip_options, _ = pip_command.parse_args([])
session = pip_command._build_session(pip_options)
pypi = PyPIRepository(pip_options=pip_options, use_json=False, session=session)

# first observe that this returns correct constraints:
ireq = InstallRequirement.from_line('bleach==1.5.0')
pypi.get_dependencies(ireq)
{<InstallRequirement object: six (from bleach==1.5.0) editable=False>,
 <InstallRequirement object: html5lib!=0.9999,!=0.99999,<0.99999999,>=0.999 (from bleach==1.5.0) editable=False>}

# I added the following breakpoint at pipenv/patched/piptools/resolver.py:200
            if str(best_match.req) == 'bleach==1.5.0':
                import ipdb; ipdb.set_trace()

# then ran the resolver:
ireq = InstallRequirement.from_line('tensorflow==1.3.0')
resolver = Resolver(constraints=[ireq], repository=pypi)
resolver.resolve()

# stepping through we fail this check, showing that 'html5lib' is already in the cache
    if ireq not in self.dependency_cache:

# so next it fetches dep from cache:
ipdb> pp self.dependency_cache[ireq]
['html5lib!=1.0b1,!=1.0b2,!=1.0b3,!=1.0b4,!=1.0b5,!=1.0b6,!=1.0b7,!=1.0b8,>=0.99999999pre',
 'six']
# aha!
ipdb> self.dependency_cache._cache_file
'/Users/anentropic/Library/Caches/pipenv/depcache-py3.6.json'

Somehow the bogus html5lib>=0.99999999pre requirement had got stuck in the dependency cache file.

Now I knew what to do…

pipenvtemp-py3 $ rm /Users/anentropic/Library/Caches/pipenv/depcache-py3.6.json
pipenvtemp-py3 $ pipenv lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project.
Locking [dev-packages] dependencies…
Locking [packages] dependencies…

Success - lock file was created without error.

This also explains why it worked under Python 2.7 - I assume because pipenv has a separate dependency cache file per python version.