bad interaction: entrypoint script vs 'setup.py develop'

I noticed a bad interaction today. In my “magic-wormhole” project (which uses versioneer), I created a new virtualenv and ran “pip install -e .”. The project uses an entrypoint, and it wrote the entrypoint script into venv/bin/wormhole. The generated script looks like:

#!/Users/warner/stuff/tahoe/magic-wormhole/ve/bin/python2.7
# EASY-INSTALL-ENTRY-SCRIPT: 'magic-wormhole==0.2.0+22.g894da44.dirty','console_scripts','wormhole'
__requires__ = 'magic-wormhole==0.2.0+22.g894da44.dirty'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('magic-wormhole==0.2.0+22.g894da44.dirty', 'console_scripts', 'wormhole')()
    )

This works fine until I make a commit, which changes the version that setup.py version reports into something different than the one frozen into the entrypoint script. After making a commit, running the script gives me an error:

pkg_resources.DistributionNotFound: The 'magic-wormhole==0.2.0+22.g894da44.dirty' distribution was not found and is required by the application

That makes me sad: if I want to use the virtualenv for testing the executable, I have to re-run pip install -e . after basically every commit.

I think one workaround might be to avoid entrypoints and just use a regular scripts= argument (in setup.py), with a small executable that does a plain import-and-run. I haven’t been able to come up with a better idea.

[warner]

I think I understand this a bit better now.

• pip install -e . calls setup.py egg_info, then builds the entrypoint scripts (among other installation tasks)
  • setup.py egg_info uses Versioneer to compute the version (once), then writes the result into SRCDIR/PKG.egg-info/PKG-INFO
• that computed version is written into the entrypoint scripts, in two significant places (__requires__ and the first argument to pkg_resources.load_entry_point()).
  • when pkg_resources is imported, it scans for __requires__ in the top-level __main__ namespace, and bails if anything it wants is unavailable
  • load_entry_point() uses the packagename/version to find the right egg-info/entry_points.txt, to import the right module

If the egg_info (SRCDIR/PKG.egg-info.PKG-INFO) version doesn’t match the generated entrypoint script’s version, you get the error.

The trick is that a local development install (setup.py develop or pip install -e .) will update the egg_info in the source tree, and regenerate the entrypoint scripts in the install directory. So multiple install directories might cause you problems:

• virtualenv ve2; ve2/bin/pip install -e .; do some tests
• make a commit, or dirty the tree
• realize you should test against python3 too
• pyvenv-3.5 ve2; ve3/bin/pip install -e .

Now the ve2/bin/SCRIPT has the old version baked in, but second pip install -e rewrote the source tree’s egg_info to have the new version. ve3/bin/SCRIPT will work, but not ve2/bin/SCRIPT.

Another common pattern that will break the local install is to run pip install .../path/to/tree from an unrelated directory: say a separate virtualenv for testing a downstream application against your development version of the Versioneer-controlled library. The install will, as a side-effect, update the egg_info in the library’s source tree, leaving the other entrypoint scripts behind.

Even if you only have a single virtualenv, there are other commands that will run setup.py egg_info as a side-effect (register, sdist, and test being the most surprising ones). So you might take a break from your testing to make a new release, and the act of creating the sdist tarball changes the egg_info but not the virtualenv’s scripts.

So my new workaround is:

• when using multiple (py2/py3) virtualenvs for a single project, do a pip install -e . in all of them at the same time
• if I ever need to run other setup.py or pip commands using that source tree, re-install-e in all virtualenvs afterwards

This seems to be ok. (incidentally, I think the scripts= entrypoints got a version baked in too, so that workaround didn’t actually work)

If anyone can think of a better fix, I’m all ears, but I suspect this is going to have to be resolved with a docs change, rather than code.

[effigies]

Cannot reproduce with recent setuptools.