Network timeout can cause failure in harvest validation

This caused CI failure here: https://dev.azure.com/dnceng/internal/_build/results?buildId=389955&view=logs&jobId=4d20224c-9cdf-5de6-b89c-4faeb3acb47a&taskId=676da2a5-a4fd-55c6-414f-028313647323&lineStart=1787&lineEnd=1901&colStart=60&colEnd=132

Relevant portion of stack:

D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018: System.AggregateException: One or more errors occurred. (One or more errors occurred. (Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..)) (One or more errors occurred. (The download of 'https://dotnetfeed.blob.core.windows.net/dotnet-core/registration/system.composition.convention/index.json' timed out because no data was received for 60000ms.))
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:  ---> System.AggregateException: One or more errors occurred. (Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:  ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:  ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    --- End of inner exception stack trace ---
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Net.Http.HttpConnection.ReadAsync(Memory`1 destination)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Net.Http.HttpConnection.ContentLengthReadStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.IO.Compression.DeflateStream.FinishReadAsyncMemory(ValueTask`1 readTask, Memory`1 buffer, CancellationToken cancellationToken)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at NuGet.Protocol.TimeoutUtility.StartWithTimeout[T](Func`2 getTask, TimeSpan timeout, String timeoutMessage, CancellationToken token)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at NuGet.Protocol.DownloadTimeoutStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.IO.Stream.CopyToAsyncInternal(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at NuGet.Protocol.HttpCacheUtility.CreateCacheFileAsync(HttpCacheResult result, HttpResponseMessage response, Action`1 ensureValidContents, CancellationToken cancellationToken)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at NuGet.Protocol.HttpSource.<>c__DisplayClass14_0`1.<<GetAsync>b__0>d.MoveNext()
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018: --- End of stack trace from previous location where exception was thrown ---
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at NuGet.Common.ConcurrencyUtilities.ExecuteWithFileLockedAsync[T](String filePath, Func`2 action, CancellationToken token)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at NuGet.Protocol.HttpSource.GetAsync[T](HttpSourceCachedRequest request, Func`2 processAsync, ILogger log, CancellationToken token)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at NuGet.Protocol.RegistrationUtility.LoadRanges(HttpSource httpSource, Uri registrationUri, String packageId, VersionRange range, SourceCacheContext cacheContext, ILogger log, CancellationToken token)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at NuGet.Protocol.RegistrationResourceV3.GetPackageMetadata(String packageId, VersionRange range, Boolean includePrerelease, Boolean includeUnlisted, SourceCacheContext cacheContext, ILogger log, CancellationToken token)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at NuGet.Protocol.RegistrationResourceV3.GetPackageMetadata(String packageId, Boolean includePrerelease, Boolean includeUnlisted, SourceCacheContext cacheContext, ILogger log, CancellationToken token)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at NuGet.Protocol.PackageMetadataResourceV3.GetMetadataAsync(String packageId, Boolean includePrerelease, Boolean includeUnlisted, SourceCacheContext sourceCacheContext, ILogger log, CancellationToken token)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at Microsoft.DotNet.Build.Tasks.Packaging.NuGetUtility.<>c__DisplayClass0_0.<GetAllVersionsForPackageId>b__0(PackageSource packageSource) in /_/src/Microsoft.DotNet.Build.Tasks.Packaging/src/NuGetUtility.cs:line 28
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Threading.Tasks.Parallel.<>c__DisplayClass33_0`2.<ForEachWorker>b__0(Int32 i)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018: --- End of stack trace from previous location where exception was thrown ---
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Threading.Tasks.Parallel.<>c__DisplayClass19_0`1.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Threading.Tasks.TaskReplicator.Replica`1.ExecuteAction(Boolean& yieldedBeforeCompletion)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Threading.Tasks.TaskReplicator.Replica.Execute()
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    --- End of inner exception stack trace ---
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 action, ParallelOptions options, Boolean stopOnFirstFailure)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018: --- End of stack trace from previous location where exception was thrown ---
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IList`1 list, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, Action`1 body)
D:\a\1\s\pkg\test\testPackages.proj(217,5): error MSB4018:    at Microsoft.DotNet.Build.Tasks.Packaging.NuGetUtility.GetAllVersionsForPackageId(String packageId, Boolean includePrerelease, Boolean includeUnlisted, Log log, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Build.Tasks.Packaging/src/NuGetUtility.cs:line 45

We should add a retry here: https://github.com/dotnet/arcade/blob/0ec2cf014f7043659cbcfb4c746879b0ac09e491/src/Microsoft.DotNet.Build.Tasks.Packaging/src/NuGetUtility.cs#L28-L41

[joperezr]

Actually I totally forgot about this as well and we haven’t seen the failure again in release because we don’t have that many PRs on release branch, Nevertheless it would probably be good to port so I’ll put up a PR.

[ericstj]

Yes to release/3.x /cc @markwilkie