pip install fails with `ValueError: too many values to unpack`

Environment

• pip version: 19.0
• Python version: 2.7.15
• OS: Linux/Amazon EMR/Amazon Linux AMI

Description

In some situations after installing packages from a requirements file this error is thrown:

Exception:
Traceback (most recent call last):
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 393, in run
    use_user_site=options.use_user_site,
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/req/__init__.py", line 57, in install_given_reqs
    **kwargs
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 919, in install
    use_user_site=use_user_site, pycompile=pycompile,
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 445, in move_wheel_files
    warn_script_location=warn_script_location,
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/wheel.py", line 583, in move_wheel_files
    outrows = get_csv_rows_for_installed(reader)
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/wheel.py", line 565, in get_csv_rows_for_installed
    for fpath, digest, length in old_csv_rows:
ValueError: too many values to unpack

Expected behavior

Install works How to Reproduce

I am not completely sure yet… I am installing from a requirements file with pre-built packages like: sudo /home/hadoop/venv2/bin/python -m pip install -r /home/hadoop/emr.requirements.txt --find-links=/home/hadoop/packages Pre-built packages are in /home/hadoop/packages

Output

...
...
Installing collected packages: alabaster, SQLAlchemy, MarkupSafe, Mako, python-editor, six, python-dateutil, alembic, chardet, idna, certifi, urllib3, requests, algoliasearch, wrapt, lazy-object-proxy, astroid, avro, colorama, jmespath, docutils, botocore, futures, s3transfer, pyasn1, rsa, PyYAML, awscli, pytz, Babel, backports.functools-lru-cache, beautifulsoup4, Jinja2, numpy, singledispatch, backports-abc, tornado, bokeh, boto3, brewer2mpl, bz2file, cgroupspy, click, configparser, confluent-kafka, coverage, pycparser, cffi, enum34, asn1crypto, ipaddress, cryptography, cycler, dask, scipy, scikit-learn, multipledispatch, cloudpickle, dask-glm, pandas, toolz, dask-searchcv, dask-ml, distance, text-unidecode, Faker, factory-boy, future, fasttext, kiwisolver, subprocess32, pyparsing, matplotlib, fbprophet, pycodestyle, mccabe, pyflakes, flake8, fuzzywuzzy, Geohash, boto, smart-open, gensim, geographiclib, geotext, patsy, statsmodels, ggplot, psutil, gnupg, graphviz, hiredis, html2text, webencodings, html5lib, imagesize, inflection, isort, jellyfish, kafka-python, langdetect, langid, ipython-genutils, decorator, traitlets, simplegeneric, backports.shutil-get-terminal-size, scandir, pathlib2, pickleshare, Pygments, ptyprocess, pexpect, wcwidth, prompt-toolkit, IPython, line-profiler, lxml, mechanize, memory-profiler, funcsigs, pbr, mock, more-itertools, xmltodict, ecdsa, pycryptodome, python-jose, pyaml, jsonpickle, aws-xray-sdk, werkzeug, backports.weakref, backports.tempfile, cookies, responses, backports.ssl-match-hostname, docker-pycreds, websocket-client, docker, jsondiff, moto, msgpack, MySQL-python, nameparser, neo4j-driver, ner, networkx, nltk, ordereddict, olefile, Pillow, planout, plotly, prettytable, psycopg2, py, pyarrow, sasl, thrift, thrift-sasl, PyHive, pylint, bcrypt, pynacl, paramiko, pysftp, pyspark-dist-explore, PyStemmer, pytest, python-dotenv, python-Levenshtein, redis, regex, LEPL, rfc6266-parser, rpy2, s3fs, seaborn, slackclient, snowballstemmer, msgpack-numpy, murmurhash, cymem, preshed, cytoolz, plac, tqdm, dill, pathlib, thinc, ujson, spacy, Sphinx, sqlparse, Tempita, sqlalchemy-migrate, sqlalchemy-redshift, tabulate, textblob, textstat, timeout-decorator, PyJWT, twilio, unicodecsv, ua-parser, user-agents, virtualenv, virtualenv-clone, stevedore, virtualenvwrapper, wget, wordcloud, xlrd, xgboost
  Found existing installation: numpy 1.16.0
    Uninstalling numpy-1.16.0:
      Successfully uninstalled numpy-1.16.0
Exception:
Traceback (most recent call last):
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 393, in run
    use_user_site=options.use_user_site,
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/req/__init__.py", line 57, in install_given_reqs
    **kwargs
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 919, in install
    use_user_site=use_user_site, pycompile=pycompile,
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 445, in move_wheel_files
    warn_script_location=warn_script_location,
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/wheel.py", line 583, in move_wheel_files
    outrows = get_csv_rows_for_installed(reader)
  File "/home/hadoop/venv2/local/lib/python2.7/site-packages/pip/_internal/wheel.py", line 565, in get_csv_rows_for_installed
    for fpath, digest, length in old_csv_rows:
ValueError: too many values to unpack

[ohemorange]

My inclination at this point would be to return to the previous (non-strict) behavior and discuss doing something more or different in a slower time frame.

Certbot would really, really appreciate this, as the change breaks certbot for all of our certbot-auto users.

Can we get the change reverted now, and fixed fully later? Otherwise we’ll have to patch and release a point version of Certbot ASAP, before people’s certs start expiring.

Some of our test files have commas in filenames. We plan to address this in a future release, but it seems safer all around to revert the breaking change and then fix things carefully in tandem.

[pradyunsg]

It looks like there were code changes in the commit in addition adding type annotations.

That’s on me. ISTM that I didn’t review that PR properly.

I’ll look into this and file a bugfix PR.