pip unexpectedly not installing latest version of git package with branch/commit pinning

(I’m posting this as a bug since nobody answered to http://stackoverflow.com/questions/30104748/pip-unexpectedly-not-installing-latest-version-of-git-package-with-branch-commit, so I think it is a bug, sorry if it isn’t.)

I have a requirements.txt file with the following line (among others):

git+https://github.com/lead-ratings/sexmachine.git@master#egg=SexMachine

When I do

pip install -r requirements.txt

I see

Requirement already satisfied (use --upgrade to upgrade): SexMachine from git+https://github.com/lead-ratings/sexmachine.git@master#egg=SexMachine in /home/myuser/virtual_env/lib/python2.7/site-packages (from -r requirements.txt (line 38))

And the package is not updated to the master version. Actually, it keeps some former version from PyPI I had listed in requirements.txt before.

It doesn’t work either if I specify a commit in the pinning or use the --no-cache-dir flag. I’m using pip 6.1.1.

If I use the --upgrade flag then it works. But then what is the point of the pinning? Why does it say “Requirement already satisfied” if it really isn’t? I’d expect pip to always install the version specified in the pinning.

[skolsuper]

In my opinion, this is not a good resolution of this ticket. This behaviour breaks the rule of least astonishment, because it goes against specificity. If this were CSS, the version number in setup.py would be a .class, and the tag referenced in requirements.txt would be an #id. The version number is set by the maintainer and applies to everyone that uses the package, the tag in requirements.txt is set by the user and applies only to this environment. It is much more specific.

Can you give me an example scenario where a user might target a specific commit and actually want to just keep the package they already have?

I think the software should respect the specificity in the requirements.txt file. If that’s not feasible, it should at least not fail silently and should alert the user that their requested branch/tag/commit was not respected.

FWIW, I struggled for 4 hours yesterday with a package and neither bumping the version number in my targetted branch, nor passing the -U flag worked. I still haven’t got my preferred version installed even now.

Here is my fab script that sets up the environment:

def setup_venv(build):
    with cd('/webapps/{0}'.format(APP_NAME)):
        if exists('venv'):
            sudo('rm -R venv')
        sudo('virtualenv venv')
        for key, value in ENV_VARS[build.lower()].items():
            var_string = 'export {0}="{1}"'.format(key, value)
            append('venv/bin/activate', var_string, use_sudo=True)
        with prefix('source venv/bin/activate'):
            sudo('pip install -U pip')
            sudo('pip install -U -r requirements.txt -r prod-requirements.txt')

and here is the requirement I can’t seem to install properly:

git+git://github.com/skolsuper/pybbm.git@tf-master#egg=pybb

I seem to end up with just the master branch every time. (In case you check the repo, I already tried bumping the version to 999, and rolled back that commit when it didn’t work)

[pztrick]

@dstufft I am raising this issue to your attention since you seem involved in the #2432 conversation regarding shallow Git clones as well as P/Rs (now closed) #453/#507 that are possibly relevant for a solution to this issue. Today I was burned by updating a Git checkout in requirements.txt like so:

# old:
git+git://github.com/celery/celery@d7d4c2a40ee4f482ae043f92e2a5391d395027a7#egg=celery[redis]
# new:
git+git://github.com/celery/celery@120770929f4a37c5373a378b75b5c41a99702af9#egg=celery[redis]

As discussed here, this will not re-fetch the new Git source code. The work-around stated above to pass the editable flag does work:

-e git+git://github.com/celery/celery@120770929f4a37c5373a378b75b5c41a99702af9#egg=celery[redis]

Of course… now I have a 54MB tree (.git directory) instead of a smaller 6MB tree (which will be duplicated in all my succeeding Docker images). I’d be happy to pass some other flag like -s git+git://... for a shallow-clone, but, failing that, perhaps PR #507 makes sense to know if source should be re-fetched (to resolve this issue)?