2020 resolver: get_topological_weights AssertionError

What did you want to do?

pip install --use-feature 2020-resolver sphinx==1.8.5
pip install --use-feature 2020-resolver sphinx sphinxcontrib-bibtex

After installing sphinx 1.8.5, pip freeze gives this:

alabaster==0.7.12
Babel==2.8.0
certifi==2020.6.20
chardet==3.0.4
docutils==0.16
idna==2.10
imagesize==1.2.0
Jinja2==2.11.2
MarkupSafe==1.1.1
packaging==20.4
Pygments==2.7.1
pyparsing==2.4.7
pytz==2020.1
requests==2.24.0
six==1.15.0
snowballstemmer==2.0.0
Sphinx==1.8.5
sphinxcontrib-serializinghtml==1.1.4
sphinxcontrib-websupport==1.2.4
urllib3==1.25.11

Output

ERROR: Exception:
Traceback (most recent call last):
  File "/home/jahn/usr/src/pipmaster/src/pip/_internal/cli/base_command.py", line 222, in _main
    status = self.run(options, args)
  File "/home/jahn/usr/src/pipmaster/src/pip/_internal/cli/req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "/home/jahn/usr/src/pipmaster/src/pip/_internal/commands/install.py", line 378, in run
    requirement_set
  File "/home/jahn/usr/src/pipmaster/src/pip/_internal/resolution/resolvelib/resolver.py", line 183, in get_installation_order
    weights = get_topological_weights(graph)
  File "/home/jahn/usr/src/pipmaster/src/pip/_internal/resolution/resolvelib/resolver.py", line 242, in get_topological_weights
    assert len(weights) == len(graph)
AssertionError

Additional information

Dependency tree (before error):

pipdeptree==1.0.0
  - pip [required: >=6.0.0, installed: 20.3.dev0]
Sphinx==1.8.5
  - alabaster [required: >=0.7,<0.8, installed: 0.7.12]
  - babel [required: >=1.3,!=2.0, installed: 2.8.0]
    - pytz [required: >=2015.7, installed: 2020.1]
  - docutils [required: >=0.11, installed: 0.16]
  - imagesize [required: Any, installed: 1.2.0]
  - Jinja2 [required: >=2.3, installed: 2.11.2]
    - MarkupSafe [required: >=0.23, installed: 1.1.1]
  - packaging [required: Any, installed: 20.4]
    - pyparsing [required: >=2.0.2, installed: 2.4.7]
    - six [required: Any, installed: 1.15.0]
  - Pygments [required: >=2.0, installed: 2.7.1]
  - requests [required: >=2.0.0, installed: 2.24.0]
    - certifi [required: >=2017.4.17, installed: 2020.6.20]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.10]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.11]
  - setuptools [required: Any, installed: 41.6.0]
  - six [required: >=1.5, installed: 1.15.0]
  - snowballstemmer [required: >=1.1, installed: 2.0.0]
  - sphinxcontrib-websupport [required: Any, installed: 1.2.4]
    - sphinxcontrib-serializinghtml [required: Any, installed: 1.1.4]

When installing sphinx and sphixcontrib-bibtex, sphinx is set to upgrade to 3.2.1. The error only appears when sphinx is given on the command line. When only sphinxcontrib-bibtex is installed, it works fine and sphinx is upgraded. The error appears both in 20.2.4 and master.

This is the value of weights in get_topological_weights:

{'six': 5, '<Python from Requires-Python>': 5, 'latexcodec': 4, 'pyyaml': 4, 'pybtex': 3, 'setuptools': 3, 'oset': 2, 'docutils': 3, 'pybtex-docutils': 2, 'pyparsing': 4, 'packaging': 3, 'pygments': 3, 'alabaster': 3, 'sphinxcontrib-devhelp': 3, 'sphinxcontrib-jsmath': 3, 'sphinxcontrib-serializinghtml': 3, 'markupsafe': 4, 'jinja2': 3, 'snowballstemmer': 3, 'sphinxcontrib-applehelp': 3, 'sphinxcontrib-htmlhelp': 3, 'idna': 4, 'chardet': 4, 'urllib3': 4, 'certifi': 4, 'requests': 3, 'pytz': 4, 'babel': 3, 'imagesize': 3, 'sphinxcontrib-qthelp': 3, 'sphinx': 2, 'sphinxcontrib-bibtex': 1, None: 0}

This is graph:

<pip._vendor.resolvelib.structs.DirectedGraph object at 0x7f99ccef19d0>

This is graph._vertices:

{'<Python from Requires-Python>', 'oset', None, 'idna', 'sphinxcontrib-devhelp', 'sphinxcontrib-jsmath', 'urllib3', 'sphinxcontrib-htmlhelp', 'six', 'imagesize', 'pyyaml', 'snowballstemmer', 'certifi', 'sphinxcontrib-qthelp', 'markupsafe', 'docutils', 'packaging', 'sphinxcontrib-bibtex', 'sphinx', 'pytz', 'alabaster', 'pyparsing', 'jinja2', 'sphinxcontrib-applehelp', 'babel', 'pygments', 'pybtex-docutils', 'setuptools', 'sphinxcontrib-websupport', 'sphinxcontrib-serializinghtml', 'pybtex', 'requests', 'latexcodec', 'chardet'}

sphinxcontrib-websupport is missing from weights.

This is graph._forwards:

{None: {'sphinxcontrib-bibtex', 'sphinx'}, 'sphinx': {'docutils', 'packaging', '<Python from Requires-Python>', 'pygments', 'setuptools', 'alabaster', 'sphinxcontrib-devhelp', 'sphinxcontrib-jsmath', 'sphinxcontrib-serializinghtml', 'jinja2', 'snowballstemmer', 'sphinxcontrib-applehelp', 'sphinxcontrib-htmlhelp', 'requests', 'babel', 'imagesize', 'sphinxcontrib-qthelp'}, 'sphinxcontrib-bibtex': {'pybtex', 'oset', 'pybtex-docutils', 'sphinx'}, 'alabaster': set(), 'babel': {'pytz'}, 'six': set(), 'packaging': {'six', 'pyparsing'}, 'pybtex': {'six', 'latexcodec', 'pyyaml', '<Python from Requires-Python>'}, 'pybtex-docutils': {'six', 'docutils', 'pybtex', '<Python from Requires-Python>'}, 'latexcodec': {'six', '<Python from Requires-Python>'}, 'docutils': set(), 'jinja2': {'markupsafe'}, 'imagesize': set(), 'requests': {'idna', 'chardet', 'urllib3', 'certifi'}, 'setuptools': set(), 'oset': {'setuptools'}, 'snowballstemmer': set(), 'pygments': set(), 'sphinxcontrib-jsmath': {'<Python from Requires-Python>'}, 'sphinxcontrib-applehelp': {'<Python from Requires-Python>'}, 'sphinxcontrib-htmlhelp': {'<Python from Requires-Python>'}, 'sphinxcontrib-qthelp': {'<Python from Requires-Python>'}, 'sphinxcontrib-devhelp': {'<Python from Requires-Python>'}, 'sphinxcontrib-serializinghtml': set(), 'sphinxcontrib-websupport': {'sphinxcontrib-serializinghtml'}, '<Python from Requires-Python>': set(), 'pyyaml': {'<Python from Requires-Python>'}, 'pytz': set(), 'markupsafe': set(), 'urllib3': set(), 'certifi': set(), 'chardet': set(), 'idna': set(), 'pyparsing': set()}

This is graph._backwards:

{None: set(), 'sphinx': {'sphinxcontrib-bibtex', None}, 'sphinxcontrib-bibtex': {None}, 'alabaster': {'sphinx'}, 'babel': {'sphinx'}, 'six': {'pybtex', 'packaging', 'latexcodec', 'pybtex-docutils'}, 'packaging': {'sphinx'}, 'pybtex': {'sphinxcontrib-bibtex', 'pybtex-docutils'}, 'pybtex-docutils': {'sphinxcontrib-bibtex'}, 'latexcodec': {'pybtex'}, 'docutils': {'pybtex-docutils', 'sphinx'}, 'jinja2': {'sphinx'}, 'imagesize': {'sphinx'}, 'requests': {'sphinx'}, 'setuptools': {'oset', 'sphinx'}, 'oset': {'sphinxcontrib-bibtex'}, 'snowballstemmer': {'sphinx'}, 'pygments': {'sphinx'}, 'sphinxcontrib-jsmath': {'sphinx'}, 'sphinxcontrib-applehelp': {'sphinx'}, 'sphinxcontrib-htmlhelp': {'sphinx'}, 'sphinxcontrib-qthelp': {'sphinx'}, 'sphinxcontrib-devhelp': {'sphinx'}, 'sphinxcontrib-serializinghtml': {'sphinxcontrib-websupport', 'sphinx'}, 'sphinxcontrib-websupport': set(), '<Python from Requires-Python>': {'pybtex-docutils', 'sphinx', 'sphinxcontrib-devhelp', 'sphinxcontrib-jsmath', 'sphinxcontrib-applehelp', 'pybtex', 'sphinxcontrib-htmlhelp', 'latexcodec', 'pyyaml', 'sphinxcontrib-qthelp'}, 'pyyaml': {'pybtex'}, 'pytz': {'babel'}, 'markupsafe': {'jinja2'}, 'urllib3': {'requests'}, 'certifi': {'requests'}, 'chardet': {'requests'}, 'idna': {'requests'}, 'pyparsing': {'packaging'}}

[uranusjr]

Can you check whether sphinxcontrib-websupport also end up in mapping? resolvelib already does cleanup on it, and I’d be very worried if the cleanup is not working. If not, pip should be able to work around this by changing the sanity check to

assert len(weights) == len(result.mapping) + 1  # Plus the santinel None node.

instead. I will find some time to also fix it on resolvelib’s side as well, assuming the mapping entries are correct.

[pradyunsg]

I’m not 100% clear whether that’s possible solely in resolvelib or whether it would need some help from pip, but that’s something we can work out.

This is definitely solvable purely on the resolvelib side – we’d basically need to rework the graph construction slightly to do the right thing.