Binary .local/bin/gitlint missing after update from 0.16.0 to 0.17.0

After upgrading to the latest release 0.17.0 the executable is gone. Uninstalling and installing again does not help. When installing 0.16.0 all is fine.

After installing 0.17.0 …

$ pip3 install --user -U gitlint
Collecting gitlint
  Using cached gitlint-0.17.0-py2.py3-none-any.whl (2.7 kB)
Requirement already satisfied: gitlint-core[trusted-deps]==0.17.0 in ./.local/lib/python3.6/site-packages (from gitlint) (0.17.0)
Requirement already satisfied: arrow>=1 in ./.local/lib/python3.6/site-packages (from gitlint-core[trusted-deps]==0.17.0->gitlint) (1.2.1)
Requirement already satisfied: Click>=8 in ./.local/lib/python3.6/site-packages (from gitlint-core[trusted-deps]==0.17.0->gitlint) (8.0.3)
Requirement already satisfied: sh>=1.13.0 in ./.local/lib/python3.6/site-packages (from gitlint-core[trusted-deps]==0.17.0->gitlint) (1.14.2)
Requirement already satisfied: python-dateutil>=2.7.0 in ./.local/lib/python3.6/site-packages (from arrow>=1->gitlint-core[trusted-deps]==0.17.0->gitlint) (2.8.0)
Requirement already satisfied: typing-extensions in ./.local/lib/python3.6/site-packages (from arrow>=1->gitlint-core[trusted-deps]==0.17.0->gitlint) (3.7.4.3)
Requirement already satisfied: importlib-metadata in ./.local/lib/python3.6/site-packages (from Click>=8->gitlint-core[trusted-deps]==0.17.0->gitlint) (4.8.1)
Requirement already satisfied: six>=1.5 in /usr/lib/python3.6/site-packages (from python-dateutil>=2.7.0->arrow>=1->gitlint-core[trusted-deps]==0.17.0->gitlint) (1.11.0)
Requirement already satisfied: zipp>=0.5 in ./.local/lib/python3.6/site-packages (from importlib-metadata->Click>=8->gitlint-core[trusted-deps]==0.17.0->gitlint) (3.6.0)
Installing collected packages: gitlint
Successfully installed gitlint-0.17.0
WARNING: You are using pip version 21.0.1; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

… no executable to be found

$ find .local -name *gitlint*
.local/lib/python3.6/site-packages/gitlint_core-0.17.0.dist-info
.local/lib/python3.6/site-packages/gitlint-0.17.0.dist-info

After installing 0.16.0 …

$ pip3 install --user -U gitlint==0.16.0
...

… everything is fine

$ find .local -name *gitlint*
.local/lib/python3.6/site-packages/gitlint_core-0.17.0.dist-info
.local/lib/python3.6/site-packages/gitlint
.local/lib/python3.6/site-packages/gitlint/files/gitlint
.local/lib/python3.6/site-packages/qa/test_gitlint.py
.local/lib/python3.6/site-packages/qa/__pycache__/test_gitlint.cpython-36.pyc
.local/lib/python3.6/site-packages/gitlint-0.16.0.dist-info
.local/bin/gitlint

$ pip3 --version
pip 21.0.1 from /usr/lib/python3.6/site-packages/pip (python 3.6)

[webknjaz]

The solution is to do a clean gitlint install rather than an in-place upgrade: pip uninstall gitlint; pip install gitlint

More precisely, it should be pip uninstall gitlint gitlint-core; pip install gitlint which would also take care of a semi-broken install.

At this point, I think we should close this particular issue

At Ansible, we’ve opted for documenting this in the FAQ: https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_4.html#other / https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_2.10.html#known-issues. Maybe, gitlint could do the same? Probably, we could change the issue label to docs at this point.

[webknjaz]

@jorisroovers I wish I saw this issue earlier. I know exactly what happens. We’ve seen this when we were splitting the project under ansible/ansible that was previously published as just ansible on PyPI but got renamed to ansible-base, and then ansible-core. The ansible distribution is also published but with different contents.

TL;DR this is a limitation of pip that is described at https://github.com/pypa/pip/issues/8509.

When you rename a distribution package (the name published on PyPI), but keep that importable package (the folder with Python files that’s installed under site-packages), there are two different packages that provide the same files (or partially the same). There used to be a gitlint/ directory installed into site-packages/ in gitlint==0.16.0. Now, there’s none, but a folder with the same name gitlint/ is shipped as a part of gitlint-core which is pulled in by gitlint. When you do a clean install, pip will resolve the deptree and unpack the dists properly.

When you have gitlint==0.16.0, there’s already site-packages/gitlint/ on disk. And pip knows about each file belonging to the said package. During the dependency resolution, it knows that it needs to install gitlint==0.17.0 and gitlint-core==0.17.0 but because of one being a dependency of the other, it knows that gitlint-core should go in first. So pip goes ahead and unpacks the contents of gitlint-core into site-packages/gitlint/, overwriting the existing files. At this point, pip knows that the files belong to gitlint==0.16.0 and it needs to uninstall it before writing the newer version to disk, it removes these files which got installed from gitlint-core because the gitlint==0.16.0’s metadata still says that they belong to this package. And then, it goes ahead and installs gitlint==0.17.0. This is how you end up with a broken installation. It is specific to pip upgrades, but not fresh installs. The issue I linked above has a number of suggestions that pip could implement to address this bug (including transactions, similar to other packaging ecosystems), but so far, nobody’s working on that.