experimental client: download functions review

Current state of _mirror_target_download() and _mirror_meta_download() in experimental client is a bit bad:

• they are actually iterators that keep returning unverified file objects to the caller, this seems unusual and surprising: I was expecting them to return a verified file object
• caller is expected to verify the contents — this makes some sense as the verification is different for e.g. root metadata and other metadata and targets… but it should be made very clear that the file object is potentially malicious data at this point
• none of the callers currently iterate these function results (they just raise if first result is invalid) so it’s not clear if mirrors actually work
• NoWorkingMirrorError can’t be raised from these iterators like it now happens (on every iteration that fails), it does not make any sense
• _mirror_target_download type annotation is wrong

I would suggest we revert to a known working design if possible (_mirror_target_download() could just be almost 1:1 the current clients _get_target_file() function but the metadata verification needs something else).

I’ll try to come up with something…

[jku]

Could pip solve that problem on its own?

By the way, in case I’ve not been clear about this: this is not an issue in how pip is designed, it’s an issue with how pypi hosts its files: One host (pypi.org) serves metadata and targets. These targets contain URLs that pip parses to find more host-URLs: these hosts only serve target files – currently the only host is pythonhosted.org but it could be anything.

Apart from parsing target files to find more hosts, this sounds completely reasonable: assuming that metadata and targets are on the same host (or that targets are all on one host) could be a mistake for TUF.

[trishankatdatadog]

Unfortunately pip does need multiple hosts for targets. It’s not quite the concept of mirrors (the hosts don’t have the same targets) but currently the mirrors configuration can be used for it with some wrestling

I don’t think this is a problem TUF should solve at the risk of added complexity, especially if it’s designed for only one use case in mind. Could pip solve that problem on its own?