Is there a way to preserve the order in which packages are installed?

I’m trying to create a reproducible install of my window manager using pipenv and I bump into this:

Better explanation now:

• cairocffi needs to be installed after xcffib
• pip doesn’t care in which order it installs packages
• you can try pip install cairocffi[xcb] to make sure that cairocffi pulls xcffib, but still, pip will install cairocffi first (there is no such thing as conditional install_requires based on extras_requires, even if it looks like pip implemented it at some point)
• thus, the solution is to sequentially install xcffib, then cairocffi, ensuring that no cache is used when cairocffi is built (pip install --no-cache-dir cairocffi)

What a mess. 🎉

_Originally posted by @ramnes in https://github.com/qtile/qtile/issues/994#issuecomment-497984551_

[uranusjr]

Thanks for the detailed write-up!

From my understanding to the description and the context mentioned above, the main problem is that cairocffi only needs xcffib when it is installed with [xcb]. So it does not want to declare the build dependency unconditionally since it is not needed when [xcb] is not requested.

In that case, I believe the general recommendation is to split the part that depends xcffib into a separate package, and make [xcb] add that package instead. So

• cairocffi-xcb declares dependency on xcffib both build-time and run-time.
• cairocffi does not declare dependency on xcffib anywhere directly, only on cairocffi-xcb if [xcb] is requested.

This may make the project structure more complex, but fits better into the modern Python packaging architecture, and allows cairocffi (and the subproject cairocffi-xcb) to have prebuilt wheels, since the compilation dependency no longer needs to be detected on install-time. Extras in Python packaging are designed to specify optional run-time dependencies, not feature switches (although they are quite often [mis-]used as such).

If you make an anology to pure-Python packages, all Python code in a project are always the same no matter. The same goes with compiled code—the compiled binary should be the same regardless of the extras. If you want to add code only when an extra is supplied, those code should be in the package specified as optional requirements, not in the main project.

[CharString]

To be very specific: I’m trying to use pipenv to have a reproducible installation of qtile, my window manager, including my config. qtile uses cairocffi to draw on an XCB surface. That is not core to cairocffi, but an extra: so “I want it to be” installed as cairocffi[xcb].

cairocffi[xcb] uses xcffib as a binding to XCB (a C library). In order to build with xcffib, cairocffi[xcb] needs xcffib to be present at build time as well as runtime. extras_require takes care of the latter, but as I understand it, that doesn’t take care of the build time requirement.

My first naïve try was

pipenv install xcffib
pipenv install cairocffi[xcb]
pipenv install qtile

That works the first time (and subsequent once when cached wheels are used), but not on a different machine when everything gets installed in arbitrary order from the Pipfile.lock

I’ve tried making a shim package that I can install with pipenv install . that depends on xcffib as a build time requirement and on xcffib and cairocffi[xcb]. I tried with setuptools.setup(setup_requires=['xcffib'], install_requires=['cariocffi[xcb]']) in a setup.py and with xcffib and setuptools in the [build-system] section in pyproject.toml and cairo in the setup.cfg

But as I understand it, the cairocffi[xcb] build gets done in isolation from my shim package, so xcffib isn’t present there at build time.

@tych0 please correct me if I am misrepresenting the situation. The only thing I know about cffi is that it’s the preferred way of c-binding of pypy as it doesn’t make any assumptions about reference counting, something the CPython c-api does do. I’ve never used it directly myself.