Allow custom urlopen for astropy.utils.data.download_file

The astropy.utils.data module is excellent, but restricted to ‘simple’ downloads. so I’m wondering if there is scope for a change such that users could use functions other that urlopen in download_file()?

My specific use case is for accessing files from behind an authentication layer, where we have a wrapper around urlopen that handles authentication redirects and all that.

Something like this would be great:

def download_file(remote_url, cache=False, show_progress=True,
                  opener=urllib.request.urlopen, **kwargs):

where kwargs then get passed to the opener. If the powers-that-be are open to the idea, I’m happy to work up a prototype and post a PR.

[duncanmmacleod]

I now wonder whether this might be possible just using mock:

with mock.patch('urllib.request.urlopen', side_effect=mycustomopener):
    download_file(...)

Since I seem to be the only requester of this functionality I think we can close this issue now, and I can post a PR with a proper implementation of my initial suggestion if required.

[eteq]

My position has generally been that download_data and get*file* are considered “public” because of just the reason @duncanmmacleod gave - that it’s already there and useful so why not let people use it.

That said, I am sensitive to the point that it’s “feature creep” to keep adding options to maintain. E.g., we’d rather the progress bar get replaced by tqdm instead of maintaining our own (been on the todo list for a while). But I think I’m receptive to the idea @duncanmmacleod is suggesting here of basically making it more able to use other functions, because that might actually encourage use of other tools. But as @pllim says, devil is in the details, so would need to see a PR or at least a bit more detail to be entirely sure how much maintenance burden is likely…