Support the use of `pip` in packaged apps

pip is currently deliberately removed from Briefcase-packaged apps. This is on the theory that apps should be shipping with all the code they require, and installation triggered by the end-user shouldn’t be necessary.

However, there is a limited subset of cases where it is necessary. #418 is one example where it has been requested; https://github.com/phildini/hera is another.

We should support this use case.

This will require:

• Ensuring that it is possible to include pip in the requires list and get a functioning version of pip
• Reconciling the discrepancy between the default site_packages and Briefcase’s app_packages.
• Providing an entry point that can be invoked programatically. By design, pip does not have a stable programmatic API, by design, and PyPA has indicated that the have no intention of adding one (and have actively removed APIs that made pip programatically invocable
• Reconciling how to use pip on platforms where runtime installation may be prohibited by app store guidelines.

[freakboy3742]

Thanks for those details @tlambert03.

The Linux case in particular makes me wonder if we need to take a slightly different approach, and leverage user-config directories - i.e., make sure the briefcase will look in ~/.somewhere / ~/Library/Application Support / C:\Users\me\Application Data (or whatever the platform appropriate user-configuration folder would be) as an extension of the system path. Although it’s only absolutely required for Linux, a “global installation” on MacOS or Windows could also potentially have permissions issues, and having plugins be user-specific might also be necessary/useful.

We could also potentially ship a “briefcase pip” which would be a shim that does the subprocess.run() step, wrapped in a nice API, writing to the platform appropriate directory, etc.

[tlambert03]

I think you’ve taken the right approach by not including pip up-front. And it’s already super-close to being able to support pip just by adding it to the requires list. In case it’s useful for the discussion, I’ll provide a little more background on our rationale for wanting to bundle pip, and what i found was necessary to make it work with briefcase.

rationale

We want to include pip because our app (napari) supports plugins, and we’d like to enable end-users to extend the functionality of the bundled app. In fact, it was this exact feature that led us to briefcase in the first place: I had been trying to figure out a good way to support plugins using pyinstaller bundles that was both easy for the developer and the end user. It was tough, to say the least. Even if we restrict ourselves to wheels for plugins, there is the issue of dependencies, and even if we bundle a handful of standard libraries (numpy, scipy, etc…) in our app, the dependency chain and version management is still tough. So we hoped that shipping a non-frozen environment that leverages pip would make all this easier, and in fact it has! It works great! (Side note: would be happy to hear about alternative strategies for plugins, if you have any). Not that this changes anything with regards to safety, but we never intended for the bundle end-user to use pip directly. We have a GUI that shows available plugins and lets the user install/uninstall them with a button… pip then runs in a subprocess.

getting it to work

Adding pip to the app requires list was sufficient to get a functioning version of pip (or at least I haven’t found any issues yet). I don’t think it is the responsibility of briefcase to provide a pip entry point, and I also don’t think a programmatic API is necessary… we’ve been able to achieve everything we need using something like subprocess.run([sys.executable, '-m', 'pip', 'install', ...]). The main challenges have to do with path and environment wrangling.

macOS

by default, running pip in a subprocess will install apps to appname.app/Contents/Resources/Support/lib/python3x/site-packages. That works fine for us, but we do want to be able to immediately discover newly pip-installed plugins on the first run (without restarting the app), so the trick there was to make sure the (empty) site-packages folder is created (right after running briefcase create) and included in the package. That way, it will automatically be on sys.path from the start. That was it for mac!

windows

same trick as mac applies here: which is to make sure that the site-packages folder at app\src\python\Lib\site-packages. Two additional tricks were required. First, because the msi installer removes empty directories, I created not only the site-packages folder, but also an empty stub file there, right after running briefcase create and before running build. As mentioned in #418, it was also necessary to add the line .\\\\Lib\\\\site-packages\n to the app\src\python\python3x._pth file … otherwise the sys.executable subprocess running pip install would not see the site-packages folder when determining already installed packages.

linux

because the AppImage format is read-only, we can’t pip install directly to the bundle, so here we add a designated path (something like appdirs.user_data_dir()) as a prefix in pip: subprocess.run([sys.executable, '-m, 'pip', 'install', '--no-warn-script-location', '--prefix', 'some/dir', ...]) and then, make sure that the corresponding site-packages directory underneath “some/dir” is added to sys.path in your actual program, and also added to PYTHONPATH environment variable when running pip uninstall.

With those tricks, I’m pretty happy with our bundled plugin installer/uninstaller. So I guess from the perspective of briefcase, if you wanted to, you could make sure that the lib/.../site-packages folder was included in the bundle if pip is in the list of requires (and for windows, update the _pth file). It does mean that newly installed packages don’t go into briefase’s app_packages, but that seems like it would be harder since that can’t currently be used directly as a pip --prefix.