Difference in binary builds between pip 19.3 and 19.3.1

Environment

• pip version: 19.3.1
• Python version: 3.7.4/3.7.5
• OS: Linux (alpine)

Description

We use multistage builds to make our docker image compact.

https://github.com/fossasia/open-event-server/blob/b7ba05cfaefdc257e9ae07d264044668dadffce5/Dockerfile

We are using python:3.7-alpine as base image which had pip 19.3 pre-installed. Installing dependencies with build tools like gcc and others in the initial stage and then copying them to the later stage. This is a common practice and recommended in the docker community.

We installed the build tools and requirements using apt and pip which installed correctly using this command:

pip install --install-option="--prefix=/install" setuptools && \
    LIBRARY_PATH=/lib:/usr/lib pip install --install-option="--prefix=/install" -r /requirements.txt

Then, we copy the installed libraries to the next stage using the command COPY --from=builder /install /usr/local.

Now, all the libraries installed in the previous stage should be present in the next stage. Which was the case until today.

When we built the release container, it did not have importlib_metadata and threw ModuleError.

After hours of debugging, and jumping through diffs of previously working diff environment, we found that it was not installed at all along with some other libraries. More time consuming debugging later, I found out a difference between environments. The broken environment had Python 3.7.5 and pip 19.3.1 and working one had Python 3.7.4 and pip 19.3.

A quick matrix test on Python 3.7.4 with pip 19.3.1 and Python 3.7.5 with pip 19.3 showed that indeed the problem was with pip and not Python version.

After analyzing the missing dependencies, we found out that the only ones missing were the ones that were built during the previous stage. One of which was the aforementioned importlib_metadata along with several other built libraries.

Expected behavior

pip 19.3.1 which is a patch version should not break backward compatibility and retain the behavior as 19.3.

How to Reproduce

1. Disable wheel packages like in alpine and build the libraries
2. Copy from one install to another
3. Find the library to be missing

OR

1. Use the above-linked dockerfile.
2. Build it using this command - docker build . -t eventyay/open-event-server
3. Run the shell using this command - docker run -it --entrypoint /bin/sh eventyay/open-event-server -s
4. Run pip freeze | grep import
5. Change the version from python:3.7-alpine to python:3.7.4-alpine or add pip install pip==19.3 in the first RUN step to lock pip, and repeat steps 2-4 and you’ll find the library

Output

import importlib_metadata
ModuleNotFoundError: No module named 'importlib_metadata'

[chrahunt]

@iamareebjamal, in your Dockerfile please pass an explicit --prefix=/install argument to pip itself instead of --install-options='--prefix=/install'. Pip will internally update --install-options as well as take other necessary measures to ensure installed packages respect the --prefix argument. I tested this locally and the resulting image (containing dependencies installed with pip 19.3.1) had the same dependencies as the image built with dependencies installed with 19.3. You’ll probably also want to pass --no-warn-script-location to reduce some of the noise which is irrelevant to your use case.

There was a bug fixed in #6606 (released in 19.3.1) as mentioned by @pradyunsg that now correctly builds projects that use PEP 517 without implicitly going through setuptools. This fix was causing an issue in your case because the packages that were now using PEP 517 were not picking up the prefix directory passed via --install-options.

[pradyunsg]

@iamareebjamal Hi! I’ve edited your bug report to read more easily (shorter sentences, imperative tone) and to tone down some of the wording used.

Using all caps (equivalent to shouting via text) isn’t a good move when you’re asking for help from volunteers who do this stuff because they find it fun. 😃

If you have any concerns with that, please file a new issue on pip or drop me an email on {username}@gmail.com.