Directory not empty: 'urllib3' when upgrading pip

Description

When trying to upgrade pip from 22.0.4 to 22.2.2 on debian, I get the following error during the upgrade process

ERROR: Could not install packages due to an OSError: [Errno 39] Directory not empty: 'urllib3'

afterwards pip binary (/usr/local/bin/pip) does not exist anymore

Another user online seems to have faced the same issue on alpine (for the same pip versions): https://stackoverflow.com/questions/73084494/pip-upgrading-within-the-python-alpine-image-error-could-not-install-packages

Expected behavior

Upgrade does not fail and I can use pip afterwards

pip version

22.0.4

Python version

3.10.5

OS

Debian

How to Reproduce

Using docker (for example):

FROM demisto/python3-deb:3.10.5.31797
RUN pip install --upgrade pip

Output

root@c93e6f957460:/# pip install --no-cache-dir --upgrade pip
Requirement already satisfied: pip in /usr/local/lib/python3.10/site-packages (22.0.4)
Collecting pip
  Downloading pip-22.2.2-py3-none-any.whl (2.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 185.6 kB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.0.4
    Uninstalling pip-22.0.4:
ERROR: Could not install packages due to an OSError: [Errno 39] Directory not empty: 'urllib3'

root@c93e6f957460:/# pip
bash: /usr/local/bin/pip: No such file or directory

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[abn]

I can confirm that the workaround specified in the fuse-overlayfs issue resolves this for me. In my case I added the following to the buildah action.

        with:
          ...
          extra-args: |
            --storage-opt=overlay.mount_program=

[uranusjr]

The Dockerfile works for me

$ cat Dockerfile 
FROM demisto/python3-deb:3.10.5.31797
RUN pip install --upgrade pip

$ docker build .     
[+] Building 42.7s (6/6) FINISHED                                                                                                   
 => [internal] load build definition from Dockerfile                                                                           0.0s
 => => transferring dockerfile: 110B                                                                                           0.0s
 => [internal] load .dockerignore                                                                                              0.0s
 => => transferring context: 2B                                                                                                0.0s
 => [internal] load metadata for docker.io/demisto/python3-deb:3.10.5.31797                                                    3.1s
 => [1/2] FROM docker.io/demisto/python3-deb:3.10.5.31797@sha256:7608af4311f53a6b3df2d43f9b344728cfe91195a7ac39a192dbff56344  24.5s
 => => resolve docker.io/demisto/python3-deb:3.10.5.31797@sha256:7608af4311f53a6b3df2d43f9b344728cfe91195a7ac39a192dbff563445  0.0s
 => => sha256:b41d2e1d193740ed01d296c0bf6a6aeb72501da60451735f8422318ae9e5401b 11.77MB / 11.77MB                               7.2s
 => => sha256:7608af4311f53a6b3df2d43f9b344728cfe91195a7ac39a192dbff563445e062 2.84kB / 2.84kB                                 0.0s
 => => sha256:218a5ed3ee2aabb99331c3455f5d2352823aa0ca40a1b3f7537bd20bb3f91761 10.64kB / 10.64kB                               0.0s
 => => sha256:e2be974225eda2dba71b5e519b9cef7d7958fa2bd12448b10eaecb4b95000eeb 1.08MB / 1.08MB                                 1.4s
 => => sha256:b85a868b505ffd0342a37e6a3b1c49f7c71878afe569a807e6238ef08252fcb7 31.38MB / 31.38MB                              22.4s
 => => sha256:f6edef4f72a413275f1493dee2ce8413be5a8223a378294b6e77a4c47a455922 233B / 233B                                     1.7s
 => => sha256:3968cb080aabd142a7d5a2576e5f3282083fe194b2266e852603383136ca19b4 3.17MB / 3.17MB                                 6.1s
 => => sha256:7b423f706e3394f5bb44aeecd2c2ffa545cdb6820f11b267addaa10d3cf92204 283B / 283B                                     6.6s
 => => sha256:00ebe814447aedb9d1a94f0e2f975723110f62c72c90d7b3c05f778febd2a963 684.26kB / 684.26kB                             9.6s
 => => sha256:4bf0234f78dd54edb2151d6c2f0c9d7546be6f4daf84487a4efc6c8c7e52bb81 2.75MB / 2.75MB                                 9.9s
 => => sha256:81b45aaa05cb11f0ae6b7b6eb93161f0b99e5602935877ed117c86c4c54345ad 1.51MB / 1.51MB                                12.8s
 => => sha256:f4785a894e2689709d3f26996a20dd678ec8d06f9900aa2327dc4c87b2b34fd0 329B / 329B                                    10.4s
 => => sha256:f030dbb1c43b64a5bf4d0a1eccfa25ee07b02142171a174c635c1d285f40a332 7.54MB / 7.54MB                                16.6s
 => => sha256:0257331ce78c3484f657eb9162eaf12f90d126c46f34fed281bd93d83dd68b9c 5.24kB / 5.24kB                                13.3s
 => => extracting sha256:b85a868b505ffd0342a37e6a3b1c49f7c71878afe569a807e6238ef08252fcb7                                      0.8s
 => => extracting sha256:e2be974225eda2dba71b5e519b9cef7d7958fa2bd12448b10eaecb4b95000eeb                                      0.1s
 => => extracting sha256:b41d2e1d193740ed01d296c0bf6a6aeb72501da60451735f8422318ae9e5401b                                      0.3s
 => => extracting sha256:f6edef4f72a413275f1493dee2ce8413be5a8223a378294b6e77a4c47a455922                                      0.0s
 => => extracting sha256:3968cb080aabd142a7d5a2576e5f3282083fe194b2266e852603383136ca19b4                                      0.1s
 => => extracting sha256:7b423f706e3394f5bb44aeecd2c2ffa545cdb6820f11b267addaa10d3cf92204                                      0.0s
 => => extracting sha256:00ebe814447aedb9d1a94f0e2f975723110f62c72c90d7b3c05f778febd2a963                                      0.0s
 => => extracting sha256:4bf0234f78dd54edb2151d6c2f0c9d7546be6f4daf84487a4efc6c8c7e52bb81                                      0.1s
 => => extracting sha256:81b45aaa05cb11f0ae6b7b6eb93161f0b99e5602935877ed117c86c4c54345ad                                      0.1s
 => => extracting sha256:f4785a894e2689709d3f26996a20dd678ec8d06f9900aa2327dc4c87b2b34fd0                                      0.0s
 => => extracting sha256:f030dbb1c43b64a5bf4d0a1eccfa25ee07b02142171a174c635c1d285f40a332                                      0.3s
 => => extracting sha256:0257331ce78c3484f657eb9162eaf12f90d126c46f34fed281bd93d83dd68b9c                                      0.0s
 => [2/2] RUN pip install --upgrade pip                                                                                       14.9s
 => exporting to image                                                                                                         0.1s
 => => exporting layers                                                                                                        0.1s
 => => writing image sha256:1fb2a3e312b19e2750f81a0abba77f735dbb227a586918efc662372275b2ec83                                   0.0s 

$ docker run $(docker images -q | head -n1) pip --version
pip 22.2.2 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)