[Feature Request] Caching mechanism for Universal Packages

Hello,

Universal Packages do not offer any caching mechanism AFAIK, so if you call the same command twice, it will download and then re-download the package (unlike NuGet for instance).

Is there anything on the roadmap to make this available? Because the packages are not zipped, and can contain multiple file in “root”, a built-in caching mechanism is probably quite hard to get right.

But maybe something simpler could work for now (might be also useful later): Create new lightweight API (preferably also in VSTS CLI) that returns checksum for the package (or some identifier), based on it’s name, version and feed.

Example

vsts package universal checksum`
    --instance "https://dev.azure.com/SomeOrg/" `
    --feed "SomeFeed" `
    --name "some-name" `
    --version "1.0.0"

Returns just: BE906A753CD532E5DDE5BC0BFFC144DA4E3C6AB30A0A7DE283C488F620511611, so it can be piped in scripts. Package users can then save this checksum into a file, and next time check the hash first before downloading, essentially having very simple caching.

We could add a switch to alternate between MD5 and SHA256 as well.

This would be very useful with wildcards supported in version switch, see here.

[zarenner]

1. The service can only return a JSON document, not a partial document or scalar just containing a specific property (although the CLI should support such filtering with --query as you mention).

2. Support for wildcards universal package versions is most definitely on our internal roadmap - while I can’t give an exact date, I will say that in my opinion it’s probably the most important feature that Universal Packages is currently missing. In the meantime, you could either use the officially documented APIs at https://docs.microsoft.com/en-us/rest/api/azure/devops/artifacts/artifact details to get package versions (note this API will return packages of all package types, and uses GUID-form package IDs which can be cumbersome), or there’s an undocumented API at https://pkgs.dev.azure.com/{org}/_packaging/{feed}/upack/packages/{package}/versions to get just the versions of the UPack package.
   When wildcard support does arrive, my guess is it will likely be implemented mainly in the Universal Packages client / Azure CLI. That is, I would not likely expect to be able to do GET .../somepkg/versions/1.*.0 as you describe. As such, I would suggest using the Azure CLI whenever possible. If it doesn’t work well for your scenarios for some reason, we’d love to know why.

The Universal Packages APIs under .../_packaging/... (such as the one above, and the one I provided earlier) are not documented. As mentioned before, they are considered an implementation detail to support the Universal Packages client (azure CLI). While unlikely to change significantly, they do not have the same long-term compatibility guarantees that the official APIs under .../_apis/... do. Whenever possible, I suggest using the officially documented APIs, but to again to support your scenario you may need to use the unofficial APIs since we don’t have official equivalents today.

Hopefully that helps.

[zarenner]

Hi @Eragra3,

While we are interested in adding caching to the Universal Packages client, it is not currently in our short-term plans. I unfortunately don’t have a good time estimate right now.

While it is an implementation detail, the existing Universal Packages API does actually emit a content hash already - e.g. a GET https://pkgs.dev.azure.com/{org}/_packaging/{feed}/upack/packages/{package}/versions/{version} will return a JSON object containing a property ‘SuperRootId’. This property is a (custom algorithm, not SHA256 or MD5) hash that represents the content and layout of the package. I believe the output of ‘vsts package universal…’ or ‘az artifacts package universal…’ will emit this ID as well.

Perhaps this addresses your requirements. I’ll leave this issue open until we know more specifics about a timeline for Universal Packages caching.