Inconsistent NuGet push order in .NET release publishing

Describe the bug

When new .NET service release is pushed a lot of packages are pushed to the NuGet feed. Among those packages are the workload manifests and the workload packages referenced from those manifests. There is currently no enforced order in which these assets are uploaded which can cause the manifest feed to refer to non-existent packages. This will cause temporary failures on dotnet workload install command.

Example:

Run dotnet workload install macos ios
  dotnet workload install macos ios
  shell: /bin/bash -e {0}
  env:
    DOTNET_ROOT: /Users/teamcity/.dotnet

Skip NuGet package signing validation. NuGet signing validation is not available on Linux or macOS https://aka.ms/workloadskippackagevalidation .
Updated advertising manifest microsoft.net.sdk.tvos.
Updated advertising manifest microsoft.net.sdk.maui.
Updated advertising manifest microsoft.net.sdk.maccatalyst.
Updated advertising manifest microsoft.net.sdk.ios.
Updated advertising manifest microsoft.net.workload.emscripten.
Updated advertising manifest microsoft.net.sdk.android.
Updated advertising manifest microsoft.net.sdk.macos.
Updated advertising manifest microsoft.net.workload.mono.toolchain.
Installing pack Microsoft.NET.Runtime.MonoAOTCompiler.Task version 6.0.2...
Writing workload pack installation record for Microsoft.NET.Runtime.MonoAOTCompiler.Task version 6.0.2...
Installing pack Microsoft.NET.Runtime.MonoTargets.Sdk version 6.0.2...
Writing workload pack installation record for Microsoft.NET.Runtime.MonoTargets.Sdk version 6.0.2...
Installing pack Microsoft.NETCore.App.Runtime.AOT.Cross.ios-arm version 6.0.2...
Workload installation failed. Rolling back installed packs...
Rolling back pack Microsoft.NET.Runtime.MonoAOTCompiler.Task installation...
Uninstalling workload pack Microsoft.NET.Runtime.MonoAOTCompiler.Task version 6.0.2…
Rolling back pack Microsoft.NET.Runtime.MonoTargets.Sdk installation...
Uninstalling workload pack Microsoft.NET.Runtime.MonoTargets.Sdk version 6.0.2…
Rolling back pack Microsoft.NETCore.App.Runtime.AOT.Cross.ios-arm installation...
Workload installation failed: microsoft.netcore.app.runtime.aot.osx-x64.cross.ios-arm::6.0.2 is not found in NuGet feeds https://nuget.pkg.github.com/emclient/index.json;https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json;https://api.nuget.org/v3/index.json;https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json".
Error: Process completed with exit code 1.

[marcpopMSFT]

Thanks for the detailed response @joelverhagen. Glad to know there’s a 2 minute validation window during which we can unlist.

I met with our release team and we have a low cost proposal for the next release. The plan is to publish all .NET packages excluding the manifest packages, poll for availability (which I understand they already do), and then publish the manifest packages as a separate step afterwards. There is potential delay from the polling and there is still the potential for customer impact since the dozen or so manifests would still potentially light up at different times for different customers but it would reduce that window from 30+ minutes down to a much smaller window.

If that goes well, we’ll continue with that option. If there are still issues, we can explore the option of pushing unlisted and then listing all of the manifests together. I’ll be meeting with some Maui folks later today and suggesting they follow the same pattern as their publish is a separate process from the core .NET package publish today.

[lewing]

This problem also shows up with version wildcards and transitive dependencies in our regular dev pipelines, nuget really needs a transactional publish https://github.com/dotnet/performance/issues/3164