Slow builds in 2.0 - how does --python interact with --platform?

I want to build a pex that supports Linux armv7l and x86_64, on Python versions 3.5 through 3.7.

Before pex 2.0, I used

--platform linux_armv7l --platform linux_x86_64 --python python3.5 --python python3.6 --python python3.7 --python-shebang "/usr/bin/env python3"

And that produced a pex file meeting those requirements in about ten seconds.

As of 2.0 I’m not sure what I need to do. pex seems to resolve requirements for every combination of platform and python, so if I specify all six relevant --platform options (--platform linux_armv7l-cp-35-m, --platform linux_x86_64-cp-36-m, etc) plus all three --python options it takes 200s.

For my use case I can simplify and specify just --platform any-py-<35/36/37>-none and the --pythons. That takes 90s.

Ultimately the dependencies are in this case resolved to py3-none-any wheels, so I figure I can probably get away with specifying --platform any-py-35-none and the --pythons. That takes 60s.

If I omit --python entirely with the single --platform arg, it takes about 15s - close to the pre-2.0 time. For every --python I add, including the default interpreter, it adds another ~15s.

In all these cases the actual resolved and bundled dependencies are identical. What effect does --python actually have on the pex over and above the platform specifier? Is there anything I can do to get the build time down while ensuring that I’m producing a file that will work where it needs to?

[jsirois]

A reasonable change has the above example down to 23s in he pex 2.0 case or ~10% slower. These measurements are with warm caches. There is one further obvious change for the cold cache case to parallelize installation of wheels in chroots. I’ll get out seperate PRs for both here.

[AlexHill]

You probably get this all the time but I’m consistently impressed by your responsiveness and turnaround - before I’ve even posted a proper repro case you’ve made a fix and a new release. Amazing, thankyou!

The new version is a lot faster for my use case.

Since platforms are now required to be exact, there is no need when building a multiplatform pex to specify both --python and --platform, --platform is enough.

Okay, this is what I was wondering and good to know. Under pex 2.0.2, when I specify multiple --python arguments, I get different artifacts so I wasn’t sure - but upon inspection I think the only thing that’s different is the version key under build_properties in PEX-INFO which I assume doesn’t affect anything at runtime.

Note though, --platform now requires all wheels be available for that platform pre-built

Yep, that’s all good. I have a pre-built universal pure-python PyYAML wheel which I supply with --find-links, and for everything else there are universal wheels. So I can actually just do --platform any-py-3<5,6,7>-none and be satisfied that if any transitive requirements change to require compilation, the build will fail.

As an appendix - with this change (these all produce effectively the same artifact with my dependencies):

--python python3.5 --python python3.6 --python python3.7 --platform linux_armv7l-cp-35-m --platform linux_armv7l-cp-36-m --platform linux_armv7l-cp-37-m --platform linux_x86_64-cp-35-m --platform linux_x86_64-cp-36-m --platform linux_x86_64-cp-37-m takes 70s

--python python3.5 --python python3.6 --python python3.7 --platform any-py-35-none --platform any-py-36-none --platform any-py-37-none takes 40s

--platform any-py-35-none --platform any-py-36-none --platform any-py-37-none takes 18s

And installing a single platform with no --python takes 8s.