Intersphinx to Numpy gets 403 HTTP error on CircleCI

Hello. Perhaps as a result of #15540, we start seeing 403 HTTP error when referring to Numpy doc via intersphinx in our own doc build on CircleCI. We do not see this locally, on Azure Pipelines, nor Travis CI.

The Numpy intersphinx is defined in https://github.com/astropy/sphinx-astropy/blob/040b7c8a92431e1a621e9261d6185ab704bbb33f/sphinx_astropy/conf/v1.py#L50-L51 .

Sphinx has its own default User Agent (see https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-user_agent).

It is not immediately clear why this failure only happens on CircleCI for us. Is there something you can do on your side, such as whitelisting?

Hope you can advise. Thanks!

xref astropy/astropy#9947 cc @saimn

Reproducing code example:

https://github.com/astropy/astropy

Error message:

Example log: https://circleci.com/gh/astropy/astropy/59676

Warning, treated as error:
The following HTTP Error has occurred fetching https://numpy.org/doc/1.18/: 403 (Forbidden)

Numpy/Python version information:

See log above.

[rgommers]

No worries. Same here, sorry for the inconvenience!

[rgommers]

I think we should leave this issue open

gh-15540 is the issue for that, no point keeping two issues around for this one issue.

When https://numpy.org/doc/latest/ becomes the new canonical link to the latest docs for intersphinx purposes

Yep that should be the goal. It’s a matter of fixing the release procedure so that stable link gets updated each time. @mattip, I can’t remember if we talked about this when you created https://github.com/numpy/doc/?