IERS_A_URL is down, do we need alternative to fallback?

IERS_A_URL [0] seems to be down for a while now and throws a Forbidden 403 Error.

I wonder whether it makes any sense to introduce an alternative to fall back to in cases like this? E.g. [1] seems like a good one from https://www.iers.org/IERS/EN/DataProducts/EarthOrientationData/eop.html. Or the one from the ftp site listed there.

[0] http://maia.usno.navy.mil/ser7/finals2000A.all [1] https://datacenter.iers.org/eop/-/somos/5Rgv/latest/7

[taldcroft]

But getting back on topic, adding a Travis cron job to push weekly updates of the IERS-A file and monthly updates of IERS-B into astropy-data sounds like a great idea. This is independent of the more general idea of mirrors for data.astropy.org. Along with putting IERS data onto our data server would be the plan of changing the default IERS URLs and also generally allowing for arbitrary lists of mirror sites for the function(s) that get data.

I am also really 👍 on hitting astropy-affiliated servers for testing. We would feel less guilty and have higher reliability (hopefully).

[taldcroft]

Ah I see, that’s clever. However it still requires a CGI script on the other end that is listening.

I can tell you that because of web security issues, putting a publicly accessible CGI script on a CfA server is a pain. It needs to be formally reviewed by an external auditor for security. If other institutions are not that strict then they are being naive (and this comes from actual bad experience at CfA, not just theory). On the other hand, just serving static data files is no problem. So from my perspective it is much easier to have a cron job running that pulls data and puts it into a directory for a static data server.

I am imagining a simple cron script that essentially rsyncs a confiigurable “master” site (defaulting to data.astropy.org) into a local directory that is exposed to a static server on the mirror site. Anyone can set this up really easily, so we can have lots of mirror sites and it requires no further maintenance on our part. We might have the cron script do something like access a different CGI on data.astropy.org to register itself and provide notification that it has completed the sync. This could update a list of mirror sites which would be available itself on data.astropy.org (and the mirrors!).

In this way we have a distributed network of mirrors that requires no centralized maintenance (i.e. keeping the Travis CI webhooks up to date), and mirror owners only need to serve static data instead of setting up CGI script servers (which as I said is a whole new can of worms for web security).