Stop relying on `__file__`

Description

As https://github.com/indygreg/PyOxidizer/issues/69 explains, __file__ is optional, and in environments running pip where the interpreter doesn’t define it: 💥

An easy way to see this in action is pyoxy run-python -- -m pip --version (releases are here: https://gregoryszorc.com/docs/pyoxy/main/)

Expected behavior

No response

pip version

(main)

Python version

PyOxy 0.2.0

OS

Ubuntu 20

How to Reproduce

1. Use an interpreter that doesn’t define __file__. PyOxy is one of them: https://gregoryszorc.com/docs/pyoxy/main/
2. Run pip with that interpreter

Output

Traceback (most recent call last):
  File "runpy", line 194, in _run_module_as_main
  File "runpy", line 87, in _run_code
  File "pip.__main__", line 29, in <module>
  File "pip._internal.cli.main", line 9, in <module>
  File "pip._internal.cli.autocompletion", line 10, in <module>
  File "pip._internal.cli.main_parser", line 8, in <module>
  File "pip._internal.cli.cmdoptions", line 22, in <module>
  File "pip._vendor", line 23, in <module>
NameError: name '__file__' is not defined

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

To be more explicit, I’m only in favour if someone can solve all the issues in doing this. And I’m not at all convinced that those issues are actually solveable…

In particular, I agree regarding dependencies. And I’d go further still - we need a solution that means that we don’t have to go through this whole process again before adding any new dependencies. So sorting out our current dependencies is only part of the issue. Also, I agree that asking dependencies to stop using __file__ “for pip’s sake” is unacceptable.

TBH, I’m not convinced by the arguments in the linked PyOxidizer issue. The docs say that __file__ may not be present “for certain types of module”. Pure Python files are a “type of module” that does have __file__, and I think it’s legitimate to use for that reason. Of course, if the Python code isn’t hosted on a filesystem, expecting to navigate the filesystem via __file__ won’t work, and the value of __file__ may only be of use for informational purposes. But we have libraries like importlib.resources to handle those needs in an import mechanism independent way. I’d expect PyOxidizer to support such mechanisms if they expect people to stop using __file__. And I would support educating people to use importlib.resources rather than accessing the filesystem directly, just as a general “good practice”. So if PyOxidizer works with mechanisms like that, the problem will solve itself in due course.

(Someone still needs to go through existing uses of __file__, in pip and elsewhere, and identify how to use importlib.resources instead - or if it’s not possible, identify the missing functionality and get it added to importlib.resources. This isn’t a “magic wand” that will fix everyone’s issues, it’s just a way to work on them within the context of existing approaches to the problem).

[pradyunsg]

I’m ambivalent on this and, at least, Paul is in favour of this. I’ve labelled this to say that we’re open to accepting PRs toward this. I do have a couple of concerns / caveats around this:

1. It’s contingent on all our vendored dependencies being on-board. I am not comfortable with the idea of pushing this as a constraint on pip’s dependencies as “because pip wants to do it”, so whomever talks to them would need to advocate for removing reliance __file__ on its own merits.
2. We’re not going to add additional patches to our vendored dependencies, to adapt for this.