ImportError in system pip wrappers after an upgrade

Overview

After upgrading to pip 10 or higher, many users are encountering error like

• ImportError: cannot import name 'main'
• TypeError: 'module' object is not callable
• ImportError: No module named _internal
• ModuleNotFoundError: No module named 'pip._internal'

These are caused by an incorrect attempt to upgrade pip, which has typically resulted in (parts of) multiple versions of pip being installed in the same Python installation, and those parts being incompatible.

It should be noted that these issues are invariably not problems with pip itself, but are due to incorrect use of pip, or unexpected interactions with system scripts that are not controlled by pip. So while we’ll try to help you solve your issue, this is not the “fault” of pip, and you will have to be prepared to do at least some of the debugging and fixes on your own.

You can reach out to your Python provider (eg: Linux Distro, Cloud Provider, etc) for help with this issue.

General Advice

First, some general advice. It is assumed that anyone encountering issues will have failed to follow some part of this advice. Listing these items here is not intended to imply that “it’s your fault and we won’t help”, but rather to help users to identify what went wrong, so that they can work out what to do next more easily.

1. Only ever use your system package manager to upgrade the system pip. The system installed pip is owned by the distribution, and if you don’t use distribution-supplied tools to manage it, you will hit problems. Yes, we know pip says “you should upgrade with pip install -U pip” - that’s true in a pip-managed installation, ideally distributions should patch this message to give appropriate instructions in the system pip, but they don’t. We’re working with them on this, but it’s not going to happen soon (remember, we’re looking at cases where people are upgrading old versions of pip here, so patches to new versions won’t help).
2. Never use sudo with pip. This follows on from the first point. If you think you need to use sudo, you’re probably trying to modify a distribution-owned file. See point 1.
3. Prefer to use --user. By doing this, you only ever install packages in your personal directories, and so you avoid interfering with the system copy of pip. But there are PATH issues you need to be aware of here. We’ll cover these later. Put simply, it’s possible to follow this advice, and still hit problems, because you’re not actually running the wrapper you installed as --user.

Debugging the Issue

Before trying to work out what’s going on, it’s critically important that you understand precisely what scripts you are running and what versions of pip the relevant Python interpreter is using.

First, identify the full absolute path of the executable script that you are running. That’s often in the traceback you get, but if it isn’t, you can use OS tools like which, or Python’s shutil.which to identify the right script. Watch out for your shell confusing the issue, with aliases or potentially stale hashed commands.

Once you have identified the script, make sure you can reproduce the problem using the absolute path to that script. If you can’t, you probably found the wrong script, so check again.

Second, work out which version of Python the script is using to run pip. You’ll often be able to get that from the shebang line of the script. This can often be the trickiest problem, as wrapper scripts can take many forms depending on what tool created them. In the worst case, you can simply make an intelligent guess at this point.

Once you know which Python is running pip, you can run python -m pip to invoke pip. In most cases, this will work fine, as it’s the wrapper causing the issue, and not pip itself. Running pip via python -m in this way is often a perfectly acceptable workaround for the issue (at least in the short term).

Now run python -m pip --version. This will give you the exact version and location of the installation of pip that your Python is seeing.

At this point, you’re usually done - the fundamental cause of all these problems is running a wrapper script which is written expecting to see a version of pip older than pip 10 (that’s why it imports pip.main) under a Python interpreter that sees a copy of pip that’s version 10 or later.

Fixing the Issue

The problem, of course, is fixing the issue. And that’s where you really are on your own. If you have changed your system installation, you really need to put it back the way that the distribution installed it. That may well require an uninstall and reinstall of your distribution’s pip package. Reinstalling is easy, of course - but uninstalling may require manually removing incorrect files. Or you may be able to force-reinstall with your distribution package manager, simply overwriting the incorrect files. Of course, this reverts you to the system-supplied version of pip. If you need a newer version, you should ask your distribution vendor, or use something like virtualenv to install it independently of your system packages.

It may be that you’re simply running the “wrong” wrapper script. Maybe you did a --user install of a new version of pip, but your PATH is set to run the system version of the wrapper rather than the user-local one installed with pip. In that case, you can simply fix your PATH. That’s usually the issue for people who do pip install --user --upgrade pip and get the pip.main error.

As already noted, python -m pip is a reliable workaround, at the cost of using a more verbose command to invoke pip.

Community Advice

The comments section of this issue is open for people to discuss specific problems, and to share potential solutions. Just to be clear, it’s quite likely with problems of this nature that you’ll need to modify system-supplied files or settings. You do so at your own risk. If you’re not comfortable modifying your OS, or running as root, you should seek expert advice. To put it another way, if by following suggestions given here, you break your system, you get to keep the pieces. There’s only so much that can be achieved remotely.

Also, the pip developers don’t provide any guarantees that advice in the comments on this issue is correct, or that it won’t damage your system. Again, use at your own risk.

Related Issues

The following issues have been reported which are related to this issue: #5240, #5221, #5588, #5495, #5493, #5487, #5447, #5432, #5373, #5326, #5318, #5253

[MannyC]

I followed some instructions elsewhere and ran

sudo apt install python-pip
pip install --upgrade pip

and hit upon this issue.

python -m pip uninstall pip

fixed it, returning me to the system pip

I have to say, although I don’t fully understand the complications involved, it doesn’t feel like pip should self-immolate like this.

[ErichBSchulz]

The advice above:

  Only ever use your system package manager to upgrade pip.

conflicts with the cheerful siren call:

You are using pip version 8.1.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

This seems to be some kind of tarpit 😕