RTD builds fail with numpy version RuntimeError

Problem All RTD builds fail with this error during ipynb execution:

RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd

Here’s a recent failure

Marked as blocker because we can’t release again until we can build docs.

[dsherry]

Against all odds… I was able to get a green RTD build today! 🎊

Yesterday I learned a bit more about how RTD runs builds. Through some trial-and-error searching, I was able to get the build to pass locally in the RTD docker container, by overriding the version of setuptools the RTD install process uses.

I don’t understand the root cause, why this happened now and why it didn’t affect featuretools, compose or (seemingly) other open source projects. However I have the following observations to offer:

• I noticed that as soon as numpy put out their 1.20.0rc1 pre-release on Dec 3rd, our RTD build started failing… 🤔 more details here.
• I noticed that the RTD docker container includes several prebuilt pyenvs, for python 2.7 and 3.5-3.9. The dockerfile includes calls to install numpy in most of those pyenvs with the --only-binary flag passed to pip
• These three posts had the error messages which came up during my trial and error debugging. The root cause for all of those cases was that somehow there were two versions of numpy installed, and that removing the older version fixed the issue.
• I saw that the RTD build steps were installing an old version of setuptools (41.0.1) before installing our dependencies. That’s confusing, because the dockerfile uses setuptools 45.1.0 to install numpy and other dependencies once the pyenvs are created.
• Through trial and error I was able to observe that upgrading the setuptools version to >=45.2.0 (45.1.0 would probably work, haven’t confirmed) prior to installing our package’s dependencies got rid of the errors sphinx-build was raising about a numpy/numpy-C-API version mismatch, and allowed RTD to complete successfully!!!

I’ll get a PR up shortly with the fix, and will hold my breath until that’s merged… 😆 I plan to file something with RTD as well.

[dsherry]

Next on the list to try (thanks @freddyaboulton for chatting about it):

• Try python 3.8 instead of python 3.7
• Print out pip freeze on readthedocs and diff that with pip freeze in my local docker builder, and see what if anything is different in the envs
• Try installing dependencies using conda or setuptools instead of pip
• Perhaps there are other .readthedocs.yml parameters which can be tweaked.
• For attempts at local repro, try readthedocs/build:7.0 (latest) instead of readthedocs/build:6.0 (stable) which is what I was using
• Read up more on how to get sphinx-build to work with -b readthedocs (use RTD sphinx builder), because I haven’t been able to get that to work yet in the docker container I’ve pulled down.