Support abstract files and directories

So far we built on the assumption that both target files and TUF metadata can be loaded from and written to the local filesystem. This, however, is no necessity. In a large-scale production environment (like e.g. Python warehouse, see PEP 458) the TUF repository management code (most notably repository_tool and its underlying repository_lib) can and is likely to run on a different node than where TUF metadata files or target files reside. To support distributed operation, TUF repository code needs to be updated as outlined below.

metadata files

• Provide an abstract file interface that supports at least reading and writing files, creating directories and listing files in a directory (implement this in securesystemslib).

• Provide a file service that implements the abstract file interface and performs said file operations on the local file system to be used below as default file backend (implement this in securesystemslib).

• Update TUF repository code to create a new or load an existing TUF repository, to obtain hashes and sizes of metadata files, and to persist metadata files, all using a customizable file backend.

  (repository_lib.generate_snapshot_metadata, repository_lib.write_metadata_file, repository_tool.create_new_repository, repository_tool.load_repository (**))

• Update securesystemslib code that is currently used by TUF repository code for file operations to support the use of a customizable file backend.

  (util.get_file_details, util.load_json_file, util.persist_temp_file, hash.digest_filename(**))

• Revise file existence checks (os.path.{isfile,isdir,exists}) in TUF repository code, and depending on which seems less invasive, or generally better suited, either

  • switch to a file system agnostic EAFP-style (e.g. catching IOError), or
  • allow TUF to perform these checks using a customizable file backend.

  (repository_lib.generate_targets_metadata, repository_lib.write_metadata_file, repository_lib._check_directory, repository_lib._delete_obsolete_metadata, repository_lib._load_top_level_metadata(**))

(**) (Non-exhaustive list of probably affected functions)

target files

@joshuagl and @sechkova have submitted PRs that decouple abstract targets in TUF metadata from their physical equivalents on disk. This work includes:

• removal of file existence checks in user functions that add target files to the internal TUF metadata store (#1008),

• support for a fileinfo argument on add target user functions to pass out-of-band obtained hashes and sizes of files (#1007),

• support for a use_existing_fileinfo on write metadata user functions, to user prior passed hashes and sizes instead of obtaining them by reading files on disk.

• Update TUF repository code to obtain hashes and sizes of target files using a customizable file backend. (Note that above PRs suffice to operate TUF with non-local target files, hence this sub-feature request is low-prio.)

[lukpueh]

Thanks to @joshuagl and @sechkova for implementing and adopting this feature!! ❤️ 🎉 💯

[woodruffw]

That proposed interface looks fantastic, thanks a ton @joshuagl!