SourceLink validation failures for CoreFX

While switching to use pipelines publishing in corefx, I encountered 2 failures in SourceLink before I decided to disable it in order to make progress.

Could not load file or assembly 'System.Web.Extensions, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The paging file is too small for this operation to complete. (Exception from HRESULT: 0x800705AF)
System.Management.Automation.RemoteException: Could not load file or assembly 'System.Web.Extensions, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The paging file is too small for this operation to complete. (Exception from HRESULT: 0x800705AF)
at <ScriptBlock>, D:\a\1\s\eng\common\tools.ps1: line 575
at <ScriptBlock>, <No file>: line 6

https://dev.azure.com/dnceng/internal/_build/results?buildId=297506&view=logs&jobId=0a6c679f-72be-5867-422e-acb741a068a3&taskId=d2ccdc12-7474-5a79-d9be-9757fb834354&lineStart=48&lineEnd=51&colStart=1&colEnd=36

The second one was:

Unable to load DLL 'iphlpapi.dll': Exception from HRESULT: 0x800705AF
System.Management.Automation.RemoteException: Unable to load DLL 'iphlpapi.dll': Exception from HRESULT: 0x800705AF
at <ScriptBlock>, <No file>: line 18

https://dev.azure.com/dnceng/internal/_build/results?buildId=298897&view=logs&jobId=0a6c679f-72be-5867-422e-acb741a068a3&taskId=d2ccdc12-7474-5a79-d9be-9757fb834354&lineStart=60&lineEnd=62&colStart=1&colEnd=37

I tried to repro locally by downloading the symbol packages from the build artifacts and ran the exact same version of the sourcelink validation ps1 script and didn’t get a repro, it was successful the 3 times I tried to run it.

@riarenas started taking a look to see if he found anything earlier today.

cc: @tmat @JohnTortugo @mmitche

[ericstj]

Cleaning the disk is important, but also trying to get powershell to avoid the memory growth is also important. Hacky fix for that would be a process boundary, perhaps there something with more finesse that would avoid hacks. I’m not sure how powershell roots objects & interacts with GC otherwise I’d provide some better suggestions: like buffer pooling.

[JohnTortugo]

I did some quick profiling and confirmed that we’re creating a bunch of Jobs – 100’s given for CoreFx number of assets. That caused the script to use about 2GB of memory:

I implemented something to limit the number of jobs that will be spawned to run in parallel and the memory footprint decreased to a fraction of that: