[Performance]: `GetTargetPathWithTargetPlatformMoniker` and `GetTargetPath` target post-processing can take multiple minutes on large projects.

Issue Description

On large highly interdependent projects the GetTargetPathWithTargetPlatformMoniker and GetTargetPath targets (from Microsoft.Common.CurrentVersion.targets) can end up taking multiple minutes to run. GetTargetPathWithTargetPlatformMonike doesn’t do anything expensive, it just creates a new project Item with some metadata, but it has a Returns attrribute that returns @(TargetPathWithTargetPlatformMoniker) which due to post processing in MSBuild to handle the returns, can take a long time to execute. Since the GetTargetPath target is essentially a passthrough for this value, it takes almost the exact same amount of time. For our large project we have seen each of these targets take upwards of 3.5 minutes to run.

The behavior can be seen to a lesser effect on other projects in the solution where each target takes ~20 second or ~1 minute to run.

Overall build time:

Time spent on the two targets (note: these targets are serial, not parallel, so it is almost minutes in total)

Steps to Reproduce

I’m happy to privately share a binlog of the build, but the descript here should be enough to identify the source of the problem.

Data

Call stack during 3minute hangs:

System.Collections.Immutable.dll!System.Collections.Immutable.SortedInt32KeyNode`1.Freeze+0x1a
System.Collections.Immutable.dll!System.Collections.Immutable.SortedInt32KeyNode`1.Freeze+0xbb
System.Collections.Immutable.dll!System.Collections.Immutable.SortedInt32KeyNode`1.Freeze+0xc9
System.Collections.Immutable.dll!System.Collections.Immutable.SortedInt32KeyNode`1.Freeze+0xc9
System.Collections.Immutable.dll!System.Collections.Immutable.ImmutableDictionary`2..ctor+0x7f
System.Collections.Immutable.dll!System.Collections.Immutable.ImmutableDictionary`2.Wrap+0x5d
System.Collections.Immutable.dll!System.Collections.Immutable.ImmutableDictionary`2.SetItem+0xb3
Microsoft.Build.dll!Microsoft.Build.Collections.CopyOnWritePropertyDictionary`1.Set+0x58
Microsoft.Build.dll!TaskItem.get_MetadataCollection+0x242
Microsoft.Build.dll!TaskItem.Equals+0x263
mscorlib.dll!System.Collections.Generic.GenericEqualityComparer`1.Equals+0x56
System.Core.dll!System.Collections.Generic.HashSet`1.Contains+0xda
Microsoft.Build.dll!<ExecuteTarget>d__44.MoveNext+0xd3f
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start+0x80
Microsoft.Build.dll!Microsoft.Build.BackEnd.TargetEntry.ExecuteTarget+0x79
Microsoft.Build.dll!<ProcessTargetStack>d__23.MoveNext+0x911
mscorlib.dll!System.Threading.ExecutionContext.RunInternal+0x172
mscorlib.dll!System.Threading.ExecutionContext.Run+0x15
mscorlib.dll!MoveNextRunner.Run+0x6f
mscorlib.dll!<>c.<Run>b__2_0+0x36
mscorlib.dll!System.Threading.Tasks.Task.Execute+0x47
mscorlib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal+0x18c
mscorlib.dll!System.Threading.Tasks.Task.ExecuteEntry+0xa1
Microsoft.Build.dll!DedicatedThreadsTaskScheduler.<InjectThread>b__6_0+0x78
mscorlib.dll!System.Threading.ExecutionContext.RunInternal+0x172
mscorlib.dll!System.Threading.ExecutionContext.Run+0x15
mscorlib.dll!System.Threading.ExecutionContext.Run+0x55
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart+0x55
[Unmanaged to Managed Transition]
clr.dll!DllCanUnloadNowInternal+0x10f3
clr.dll!DllCanUnloadNowInternal+0x1000
clr.dll!DllCanUnloadNowInternal+0x18b0
clr.dll!MetaDataGetDispenser+0xcdaf
clr.dll!DllCanUnloadNowInternal+0x2498
clr.dll!DllCanUnloadNowInternal+0x2403
clr.dll!DllCanUnloadNowInternal+0x2342
clr.dll!DllCanUnloadNowInternal+0x2533
clr.dll!MetaDataGetDispenser+0xcc99
clr.dll!DllCanUnloadNowInternal+0x6015
KERNEL32.dll!BaseThreadInitThunk+0x14
ntdll.dll!RtlUserThreadStart+0x21

Analysis

Example project structure:

• Service.Stuff.csproj
  • Service.Core.csproj
    • Service.Interfaces.csproj
      • Common.csproj
  • Service.Interfaces.csproj
    • Common.csproj
  • Common.csproj

GetTargetPath gets called on each of these, and because of the metadata introduced at different levels, it produces 7 different TargetPathWithTargetPlatformMoniker items. For Common.csproj, these items all have the same ItemSpec (essentially just "Common.csproj") but they differ in their metadata.

This example has 7 items. In our solution with 385 projects we have a PostBuild project that references every other project. When this runs, the @(TargetPathWithTargetPlatformMoniker) collection has ~50K items.

As part of the post-processing for the target, since they have a Returns attribute, there is some work done to dedupe this collection. It uses a HashSet<TaskItem> to do the deduping. However, the hashcode for the TaskItem only takes into account the ItemSpec so there ends up being a lot of hash collisions and it falls back to doing an expensive comparison which generates and compares the entire metadata collection of the item.

Our PostBuild project is the most extreme example of this, but you can see that this problem appears even in projects that are significantly smaller, but this step still ends up taking an excessively long time (we have multiple projects that take >10s for each of these targets and it compounds over all the projects).

Versions & Configurations

We are using MSBuild 17.2.0, but i’ve analyzed the call paths and I don’t see any changes between that version and the most recent build that would change this behavior.

Microsoft (R) Build Engine version 17.2.1+52cd2da31 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

17.2.1.25201

Regression

• yes
• no

Regression Details

We have been using this common PostBuild project for years, but apparently the slowdowns have only really started as part of modernizing to newer versions of MSBuild and .NET.

[veleek]

Feedback item created and binlog attached: https://developercommunity.visualstudio.com/t/Performance:-GetTargetPathWithTargetPl/10405139

[veleek]

Yeah, I’m not sure it makes sense to include the metadata in the general case, most because it’s not immutable (I think?).

In this de-duping scenario the HashSet isn’t persistent so mutability doesn’t matter. just using a custom comparator to override GetHashCode means that the object metadata is traversed at most once for each item in the list (assuming no hash collisions which should be rare) so it’d be good enough and avoid calls to Equals which forces the metadata enumeration for both objects.