question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Slow design-time builds for large solution

See original GitHub issue

Issue Description

Hi,

productivity in our company is heavily hindered by the slowness of design-time builds which translates into the slowness of solution load (and reload) times in the IDE (Visual Studio and Rider).

For example, we are working with solution containing about 270 C# and F# projects and use Central Package Management to manage version of our NuGet dependencies. The load time of such solution is about 2-3 minutes which is far too long to stay focused.

The fact that we use a mix of C# and F# projects is probably irrelevant because according to my internal testing the solution load time when all projects are C# projects is the same. Also, the usage of central package version management seems to be not really affecting how long it takes to run an individual design-time build. But the concerning consequence of using central package management is that any slightest change in the central file (Directory.Packages.Props) triggers a reload of all projects in the solution so the slowness of design-time builds is exacerbated by central package management.

We’ve prepared a skeleton version of our solution https://github.com/marcin-krystianc/TestSolutions/tree/master/LargeAppWithPrivatePackagesCentralisedNGBVRemoved which can be used for testing.

I don’t think that the problem is unique to our solution though, my suspicion (confirmed with some experiments) is that any solution of similar size has similar performance characteristics. So, given that the problem is relevant to all large solutions, we are wondering whether there are any plans to improve the performance of design-time builds for large solutions? Maybe a static graph feature can help? It turned out to be a huge win for NuGet restores (https://devblogs.microsoft.com/visualstudio/performance-improvements-in-nuget/#msbuild-static-graph-evaluation) so maybe it is possible to apply it for design-time builds as well? Are you able to provide any hints on where to look for any improvements? Looking at text logs or binary logs (/bl) didn’t give any obvious answers so far.

Steps to Reproduce

dotnet msbuild /t:GetSuggestedWorkloads;_CheckForInvalidConfigurationAndPlatform;ResolveReferences;ResolveProjectReferences;ResolveAssemblyReferences;ResolveComReferences;ResolveNativeReferences;ResolveSdkReferences;ResolveFrameworkReferences;ResolvePackageDependenciesDesignTime;Compile;CoreCompile ^
/p:AndroidPreserveUserData=True ^
/p:AndroidUseManagedDesignTimeResourceGenerator=True ^
/p:BuildingByReSharper=True ^
/p:BuildingProject=False ^
/p:BuildProjectReferences=False ^
/p:ContinueOnError=ErrorAndContinue ^
/p:DesignTimeBuild=True ^
/p:DesignTimeSilentResolution=False ^
/p:JetBrainsDesignTimeBuild=True ^
/p:ProvideCommandLineArgs=True ^
/p:ResolveAssemblyReferencesSilent=False ^
/p:SkipCompilerExecution=True ^
/p:TargetFramework=net5.0 ^
/v:n ^
/m:1 ^
/bl ^
/flp:v=n;PerformanceSummary ^
/clp:Summary ^
/clp:PerformanceSummary > log.txt

Data

Sample logs from the command above. Please note, that this is not exactly the same as loading the solution in IDE, because IDE runs desgn-time build for each project individually wehereas we run it for entire solution all at once:

...
     851 ms  d:\workspace\TestSolutions\LargeAppWithPrivatePackagesCentralisedNGBVRemoved\solution\Project53\Project53.fsproj  24 calls
                  0 ms  GetSuggestedWorkloads                      1 calls
                  7 ms  _CheckForInvalidConfigurationAndPlatform   1 calls
                600 ms  ResolveReferences                          1 calls
                  2 ms  GetTargetFrameworks                        6 calls
                  1 ms  GetTargetPath                              6 calls
                  0 ms  ResolveProjectReferences                   1 calls
                  0 ms  ResolveAssemblyReferences                  1 calls
                  0 ms  ResolveComReferences                       1 calls
                  0 ms  ResolveNativeReferences                    1 calls
                  0 ms  ResolveSdkReferences                       1 calls
                  0 ms  ResolveFrameworkReferences                 1 calls
                161 ms  ResolvePackageDependenciesDesignTime       1 calls
                 79 ms  Compile                                    1 calls
                  0 ms  CoreCompile                                1 calls
     1001 ms  d:\workspace\TestSolutions\LargeAppWithPrivatePackagesCentralisedNGBVRemoved\solution\Project12\Project12.fsproj  15 calls
                  0 ms  GetSuggestedWorkloads                      1 calls
                  7 ms  _CheckForInvalidConfigurationAndPlatform   1 calls
                791 ms  ResolveReferences                          1 calls
                  3 ms  GetTargetFrameworks                        1 calls
                  0 ms  GetTargetFrameworksWithPlatformForSingleTargetFramework   1 calls
                  0 ms  GetTargetPath                              1 calls
                  0 ms  ResolveProjectReferences                   1 calls
                  0 ms  ResolveAssemblyReferences                  1 calls
                  0 ms  ResolveComReferences                       1 calls
                  0 ms  ResolveNativeReferences                    1 calls
                  0 ms  ResolveSdkReferences                       1 calls
                  0 ms  ResolveFrameworkReferences                 1 calls
                127 ms  ResolvePackageDependenciesDesignTime       1 calls
                 70 ms  Compile                                    1 calls
                  0 ms  CoreCompile                                1 calls
     1392 ms  d:\workspace\TestSolutions\LargeAppWithPrivatePackagesCentralisedNGBVRemoved\solution\Project276\Project276.csproj  12 calls
                  0 ms  GetSuggestedWorkloads                      1 calls
                  5 ms  _CheckForInvalidConfigurationAndPlatform   1 calls
               1113 ms  ResolveReferences                          1 calls
                  0 ms  ResolveProjectReferences                   1 calls
                  0 ms  ResolveAssemblyReferences                  1 calls
                  0 ms  ResolveComReferences                       1 calls
                  0 ms  ResolveNativeReferences                    1 calls
                  0 ms  ResolveSdkReferences                       1 calls
                  0 ms  ResolveFrameworkReferences                 1 calls
                198 ms  ResolvePackageDependenciesDesignTime       1 calls
                 76 ms  Compile                                    1 calls
                  0 ms  CoreCompile                                1 calls
     3875 ms  d:\workspace\TestSolutions\LargeAppWithPrivatePackagesCentralisedNGBVRemoved\solution\Project47\Project47.fsproj  26 calls
                  1 ms  GetSuggestedWorkloads                      1 calls
                113 ms  _CheckForInvalidConfigurationAndPlatform   1 calls
               3426 ms  ResolveReferences                          1 calls
                  2 ms  GetTargetFrameworks                        7 calls
                  1 ms  GetTargetPath                              7 calls
                  0 ms  ResolveProjectReferences                   1 calls
                  0 ms  ResolveAssemblyReferences                  1 calls
                  0 ms  ResolveComReferences                       1 calls
                  0 ms  ResolveNativeReferences                    1 calls
                  0 ms  ResolveSdkReferences                       1 calls
                  0 ms  ResolveFrameworkReferences                 1 calls
                224 ms  ResolvePackageDependenciesDesignTime       1 calls
                105 ms  Compile                                    1 calls
                  0 ms  CoreCompile                                1 calls
    125614 ms  d:\workspace\TestSolutions\LargeAppWithPrivatePackagesCentralisedNGBVRemoved\solution\LargeAppWithPrivatePackagesCentralisedNGBVRemoved.sln   1 calls
              125614 ms  GetSuggestedWorkloads;_CheckForInvalidConfigurationAndPlatform;ResolveReferences;ResolveProjectReferences;ResolveAssemblyReferences;ResolveComReferences;ResolveNativeReferences;ResolveSdkReferences;ResolveFrameworkReferences;ResolvePackageDependenciesDesignTime;Compile;CoreCompile   1 calls

Target Performance Summary:
        0 ms  ValidateProjects                           1 calls
        0 ms  ValidateToolsVersions                      1 calls
        2 ms  GenerateMSBuildEditorConfigFile          155 calls
        2 ms  InitializeSourceControlInformation       276 calls
        3 ms  ResolvePackageDependenciesForBuild       276 calls
        3 ms  GetReferenceAssemblyPaths                276 calls
        3 ms  AddSourceRevisionToInformationalVersion  276 calls
        3 ms  GenerateAssemblyInfo                     276 calls
        3 ms  GetTargetPath                            212 calls
        3 ms  SetWin32ManifestProperties               276 calls
        4 ms  AfterCompile                             276 calls
        4 ms  PrepareProjectReferences                 276 calls
        4 ms  GetFrameworkPaths                        276 calls
        5 ms  _SetTargetFrameworkMonikerAttribute      155 calls
        5 ms  BeforeResolveReferences                  276 calls
        6 ms  _ReportUpgradeNetAnalyzersNuGetWarning   155 calls
        6 ms  AfterResolveReferences                   276 calls
        6 ms  EnableIntermediateOutputPathMismatchWarning 276 calls
        7 ms  BeforeCompile                            276 calls
        7 ms  _AddOutputPathToGlobalPropertiesToRemove 276 calls
        8 ms  _BeforeVBCSCoreCompile                   155 calls
        8 ms  _CheckForUnsupportedHostingUsage         276 calls
        9 ms  _GetProjectJsonPath                      276 calls
        9 ms  _DefaultMicrosoftNETPlatformLibrary      276 calls
        9 ms  CollectPackageReferences                 276 calls
       10 ms  _CheckForUnsupportedAppHostUsage          70 calls
       10 ms  ExpandSDKReferences                      276 calls
       10 ms  ResolveLockFileAnalyzers                 276 calls
       11 ms  _CheckForObsoleteDotNetCliToolReferences 276 calls
       12 ms  ValidateSolutionConfiguration              1 calls
       12 ms  ValidateCommandLineProperties            276 calls
       13 ms  AddImplicitDefineConstants               276 calls
       14 ms  _CheckForUnsupportedCppNETCoreVersion    276 calls
       14 ms  _CheckForUnsupportedNETCoreVersion       276 calls
       16 ms  _CheckForLanguageAndFeatureCombinationSupport 276 calls
       18 ms  GenerateProgramFile                       70 calls
       18 ms  GenerateMSBuildEditorConfigFileShouldRun 155 calls
       19 ms  CollectFSharpDesignTimeTools             121 calls
       23 ms  GetTargetPathWithTargetPlatformMoniker   212 calls
       27 ms  _ComputePackageReferencePublish          276 calls
       28 ms  _ComputeSkipAnalyzers                    155 calls
       35 ms  _ComputeTargetFrameworkItems             144 calls
       49 ms  AddGlobalAnalyzerConfigForPackage_MicrosoftCodeAnalysisNetAnalyzers 155 calls
       51 ms  _GenerateCompileInputs                   276 calls
       55 ms  GetAssemblyVersion                       276 calls
       58 ms  _CollectTargetFrameworkForTelemetry      276 calls
       58 ms  CheckForImplicitPackageReferenceOverrides 276 calls
       62 ms  _SetEmbeddedWin32ManifestProperties      276 calls
       68 ms  GetTargetFrameworks                      212 calls
       72 ms  GetAssemblyAttributes                    276 calls
       84 ms  GenerateFSharpTextResources              121 calls
       86 ms  GetTargetFrameworksWithPlatformForSingleTargetFramework 212 calls
       91 ms  ResolveSDKReferences                     277 calls
      109 ms  ApplyImplicitVersions                    276 calls
      111 ms  _GetRestoreProjectStyle                  276 calls
      118 ms  CheckForDuplicateItems                   276 calls
      121 ms  UpdateAspNetToFrameworkReference         276 calls
      124 ms  IncludeTransitiveProjectReferences       276 calls
      131 ms  _GetFrameworkAssemblyReferences          121 calls
      142 ms  ValidateExecutableReferences             206 calls
      143 ms  GenerateMSBuildEditorConfigFileCore      155 calls
      195 ms  PrepareForBuild                          276 calls
      216 ms  ResolveNativeReferences                    1 calls
      240 ms  ResolveComReferences                       1 calls
      288 ms  GenerateNETCompatibleDefineConstants     276 calls
      293 ms  CreateGeneratedAssemblyInfoInputsCacheFile 276 calls
      337 ms  CoreGenerateAssemblyInfo                 276 calls
      348 ms  GenerateTargetFrameworkMonikerAttribute  276 calls
      420 ms  ResolveFrameworkReferences               277 calls
      494 ms  GetTargetFrameworksWithPlatformFromInnerBuilds 144 calls
      662 ms  RedirectTPReferenceToNewRedistributableLocation 121 calls
      733 ms  _GenerateCompileDependencyCache          121 calls
      735 ms  ProcessFrameworkReferences               276 calls
      760 ms  ResolveLockFileCopyLocalFiles            276 calls
      939 ms  ResolveTargetingPackAssets               276 calls
     1044 ms  RedirectFSharpCoreReferenceToNewRedistributableLocation 121 calls
     1123 ms  ResolveLockFileReferences                276 calls
     1146 ms  ResolveAssemblyReferencesDesignTime      276 calls
     1527 ms  AssignProjectConfiguration               276 calls
     2019 ms  _SplitProjectReferencesByFileExistence   276 calls
     2084 ms  _CheckForInvalidConfigurationAndPlatform 277 calls
     4153 ms  ResolvePackageAssets                     276 calls
     4199 ms  CoreCompile                              277 calls
     5660 ms  ResolveProjectReferences                 277 calls
     7393 ms  FindReferenceAssembliesForReferences     276 calls
     8683 ms  GetSuggestedWorkloads                      1 calls
     9185 ms  _HandlePackageFileConflicts              276 calls
    11768 ms  _GetProjectReferenceTargetFrameworkProperties 276 calls
    14755 ms  Compile                                  277 calls
    18153 ms  ResolveAssemblyReferences                277 calls
    31281 ms  RunResolvePackageDependencies            276 calls
    40710 ms  ResolvePackageDependenciesDesignTime     277 calls
    60595 ms  ResolveReferences                        277 calls

Task Performance Summary:
        3 ms  Message                                    1 calls
       12 ms  GetFrameworkPath                         276 calls
       13 ms  FSharpEmbedResXSource                    121 calls
       17 ms  FSharpEmbedResourceText                  121 calls
       24 ms  GetAssemblyVersion                       276 calls
       24 ms  FindAppConfigFile                        276 calls
       29 ms  AllowEmptyTelemetry                      276 calls
       31 ms  CheckForDuplicateFrameworkReferences     276 calls
       31 ms  CombineXmlElements                       212 calls
       32 ms  CheckForImplicitPackageReferenceOverrides 276 calls
       33 ms  CombineTargetFrameworkInfoProperties     212 calls
       45 ms  ApplyImplicitVersions                    276 calls
       48 ms  ResolveFrameworkReferences               276 calls
       62 ms  GenerateMSBuildEditorConfig              155 calls
       67 ms  GetRestoreProjectStyleTask               276 calls
       69 ms  ResolveAppHosts                          276 calls
       72 ms  CheckForDuplicateItems                   828 calls
       83 ms  CheckIfPackageReferenceShouldBeFrameworkReference 552 calls
      124 ms  ValidateExecutableReferences             206 calls
      138 ms  Hash                                     397 calls
      153 ms  GetPackageDirectory                      2208 calls
      163 ms  MakeDir                                  397 calls
      342 ms  JoinItems                                276 calls
      343 ms  WriteLinesToFile                         397 calls
      534 ms  ProcessFrameworkReferences               276 calls
      858 ms  ResolveTargetingPackAssets               276 calls
     1331 ms  GetReferenceNearestTargetFrameworkTask   304 calls
     1341 ms  AssignProjectConfiguration               276 calls
     1350 ms  Fsc                                      121 calls
     3434 ms  PreprocessPackageDependenciesDesignTime  276 calls
     3854 ms  ResolvePackageAssets                     276 calls
     7038 ms  ResolvePackageFileConflicts              276 calls
    16436 ms  ResolveAssemblyReference                 276 calls
    30768 ms  ResolvePackageDependencies               276 calls
    139515 ms  MSBuild                                  736 calls

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:02:06.04

Analysis

  • Using higher parallelism (/m) doesn’t really make it much faster as it seems that msbuild utilised more machine resources, but at the same time it also does much more work as caching inside msbuild is less effective.

Versions & Configurations

  • dotnet 6.0.300
  • VS 17.2.3

Regression?

I don’t think so.

Issue Analytics

  • State:closed
  • Created a year ago
  • Reactions:3
  • Comments:17 (15 by maintainers)

github_iconTop GitHub Comments

1reaction
dsplaistedcommented, Sep 7, 2022

Looking at this more, it looks like ResolvePackageDependencies does normally run, but it does less work unless EmitLegacyAssetsFileItems is true.

However, it still has to read the assets file. It uses an in-memory cache for this so it won’t read the same assets file multiple times in the same build, but it looks like this may still be a perf hit in this scenario with a lot of projects in the solution.

Most of the logic for reading the assets file is now in ResolvePackageAssets, which maintains its own optimized cache of the results and won’t read the assets file at all unless the cache is out of date. So we may want to consider reading the PackageDefinitions and PackageDependencies items in ResolvePackageAssets, so that ResolvePackageDependencies doesn’t need to be called at all unless EmitLegacyAssetsFileItems is true. That should improve incremental build perf and might also significantly improve the design time build perf for this large solution scenario.

@dotnet/msbuild @marcpopMSFT

0reactions
dsplaistedcommented, Dec 1, 2022

This should be fixed in 7.0.200 via #28405.

Thanks a bunch @marcin-krystianc for the detailed information which enabled us to investigate and address this!

Read more comments on GitHub >

github_iconTop Results From Across the Web

The Scourge of Design-Time Builds | Panopticon Central
For example, a common place where slow design-time builds show up is switching between configurations. If you switch a large solution, say, ...
Read more >
design-time-builds.md - dotnet/project-system
Design-time builds are special builds that are launched by the project ... For a slow design-time, look for the target performance summary at...
Read more >
Large C# and VB solutions load significantly faster in 15.5 ...
The window will show a list of targets and design time build time for each project. Identify slow projects and inspect their targets,...
Read more >
Speed up ReSharper (and Visual Studio)
ReSharper's incremental build can considerably reduce build time, especially for large solutions. Code Editing | Context Actions: here you can ...
Read more >
Improve start up time - Visual Studio (Windows)
Open the Performance Manager dialog box by choosing Help > Visual Studio Performance Manager from the menu bar. If an extension is slowing...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found