client/updater design: let user download targets?

This is a future wishlist item for client API (#1135), based on working on pip integration.

Current issues with TUF from pip point of view:

• No way to implement progress indication (a current feature) in pip
• Difficult to implement parallel downloads in pip (a potential future feature)
• No way to control low level HTTP details (authentication, timeouts, concurrency)

Some of these could be fixed by adding more complexity to TUF… but potentially the correct fix is to separate downloading from the other TUF functionality and let the clients do that themselves.

High level functionality list of the client currently looks like this:

1. generic metadata update (client.updater.Updater.refresh())
2. target metadata update (client.updater.Updater.get_one_valid_targetinfo())
3. resolving potential download URLs for target (mirrors.get_list_of_mirrors(), happens during all metadata and target downloads)
4. local cache check (client.updater.Updater.updated_targets())
5. downloading target (client.updater.Updater.download_target())
6. verifying target (happens inside client.updater.Updater.download_target())

Maybe we can optionally let user handle “downloading target” and expose “verifying target” step in a more usable way to the user. This means that “resolving download URLs” must be better exposed to user. Also “local cache check” need re-design: if user downloads files, they also control caching so we can’t expect tuf cache directory structure to exist.

[trishankatdatadog]

Anyway, the community may have its security reasons about not letting the client handle downloads on its own and I would like to hear them before implementing it. (@joshuagl @lukpueh @trishankatdatadog )

As long as we use 3rd-party code to download, we can never guarantee what it actually does.

[sechkova]

I had a look on the current implementation and I would vote for implementing minimal changes in the current Updater code trying to achieve “let user download targets” sooner and then transfer this to the client refactor.

I have some questions and comments for @jku but I’d appreciate general comments related to the client API:

let user handle “downloading target” and expose “verifying target”

I don’t see a reason why we cannot expose a verify_target function and keep the current download_target (which I would rename to a more explicit download_verified_target but this will change the current API so maybe not). Anyway, the community may have its security reasons about not letting the client handle downloads on its own and I would like to hear them before implementing it. (@joshuagl @lukpueh @trishankatdatadog )

“resolving download URLs” must be better exposed to user

I can speculate what do you mean @jku but can you be more specific 😃 Do you mean adding get_list_of_mirrors or similar to the client module or you dislike something from the function internals?

“local cache check” need re-design

Currently “updated_targets” looks for targets in a path constructed by filepath and destination_directory. Do you think this logic has to be handled by user too? Maybe an example form the pip local cache that don’t match tuf’s expectations?