[feature] Provide a hook for transparently overriding source downloads

Please provide hooks for conans.tools.get, conans.tools.download, and corresponding Conan 2 functions, so that the download handling and caching can be overridden by custom logic.

The inputs to the custom download hook would be the URL and checksum of the source archive. The hook would return a path to a local read-only file or write content to a given target path. Checksum verification and unzipping should be handled by conan. Error handling could be done by raising an exception (abort operation) or a special return code to fallback to the default download handling.

Motivation is to allow offline builds in isolated environments using a “custom local repository” for the source code archives. One requirement is that the recipes and conandata.yml are not modified and can point to the original online resources. This should not affect the reproducibility when using the recommended SHA256 checksum.

Below is an example use case of my current setup:

• Recipes are located in a local git repository. Most of the recipes have been manually downloaded from Conan Center Index and contain no or only minor modifications. Specifically, all recipes use conans.tools.get with URL and SHA256 specified in conandata.yml.
• Source code has been downloaded manually into a local folder from the URLs in the conandata.yml.
• A custom download hook has been implemented by monkey patching the conans.tools.get inside a pre_source hook. The custom download hook finds a local source file path based on the given sha256sum.
• Packages in the local .conan cache are built from source with conan create using the recipes in the local git repository, and the download hook unzips source files directly from the local download folder, so there is no network activity.

Future ideas: (not needed for my current use case)

• Provide similar hooks for scm / git clone.
• Provide hooks for replacing parts of the conan server API to allow more offline use-cases without conan_server.

The feature request is somewhat related to: #6876, #6944, #11243, and other issues about offline builds and source code caching.

• I’ve read the CONTRIBUTING guide.

[goodtune]

I have a slightly different use case to mention, and would consider it a powerful addition for corporate use.

Our company policy disallows the retrieval of source code directly from external sites, however we do have an Artifactory instance that provides the VCS Repository, that allows us to retrieve tags or branches from hosted Git services, such as Github.

As an example, the opentelemetry-cpp recipe from conan-io/conan-center-index has the following conandata.yml:

sources:
  "1.4.1":
    url: "https://github.com/open-telemetry/opentelemetry-cpp/archive/v1.4.1.tar.gz"
    sha256: "301b1ab74a664723560f46c29f228360aff1e2d63e930b963755ea077ae67524"

patches:
  "1.4.1":
    - patch_file: "patches/1.4.0-0001-fix-cmake.patch"
      base_path: "source_subfolder"

We aren’t allowed to use an HTTP proxy like Squid, but I could rewrite the URL to fetch the archive - for example "https://github.com/open-telemetry/opentelemetry-cpp/archive/v1.4.1.tar.gz" would become "https://artifactory.corp.local/artifactory/api/vcs/downloadTag/github/open-telemetry/opentelemetry-cpp/v1.4.1?ext=tar.gz". This would benefit subsequent users on our network as Artifactory can cache it forever.

The HTTP headers from this download (important ones selectively shown below) show that we’ve maintained the sha256 value.

HTTP/1.1 200 OK
Content-Type: application/x-gzip
X-Checksum-Sha256: 301b1ab74a664723560f46c29f228360aff1e2d63e930b963755ea077ae67524
X-Artifactory-Filename: opentelemetry-cpp-v1.4.1.tar.gz

I’ve just started my search for related tickets, if this has been discussed elsewhere, I can add my comments there.

[memsharded]

Our intent is to get 2.0 out before EOY, but it is a bit tight, and it also depends on the migration efforts in ConanCenter. But definitely soon. The best way to contribute atm is to test the released beta.5 and give feedback.