Discussion: Change default DebugType to embedded for .NET Core 3

Inspired from this issue and twitter thread https://github.com/dotnet/docs/pull/9110 https://twitter.com/rrelyea/status/1064587440935956482

I want to propose that the default DebugType in the SDK change from portable to embedded. I would also propose that Sourcelink be included/used by default.

At the very least, there should be one property ConfigureOpenSourceDefaults that turns these on as best practices.

My rationale is driven by the pit of success. I believe that out-of-the-box, developers should produce fully debuggable code. This is similar to how JS typically generates/includes sourcemaps by default, so there is precedent.

Few notes:

• Embedding a pdb yields a file size that’s 20-30% bigger.
• Some .NET tools, like .NET Native do not support symbol server; the symbols need to be alongside the dll
• Symbol server significantly slows down VS debug startup as it probes for files

Some things like Appx/MSIX packaging strip-out pdb files by default. They rely on extra properties (AppxPackageIncludePrivateSymbols) to include them, which I’m sure 99% of people wouldn’t know to add. Without them, the stack traces aren’t as useful.

Symbol servers may be blocked by corporate firewalls, or otherwise affected by low-bandwidth connections (think airplanes, trains, mobile hotspots, and anywhere else with slower/spotty connections).

I believe that the extra size of the files with embedded pdb’s is worth the trade-off for most cases as it presents the best debugging experience without any extra configuration. It just works.

Thanks

[NickCraver]

I want to add another angle here: I maintain an exceptions logging library, we deal with exceptions, and we get issues reported from exceptions. Stack traces are golden. They’re incredibly useful. I strongly believe default experience for any app should have stack traces include the file paths and line numbers. For this to happen the deployment platform must be able to read the symbols, and they have to be there. With SourceLink, it even means the exceptions have the git hash in the URL path, pointing directly to the exact code throwing. That’s awesome. But only if we get it. Without users having symbols in the first place it’s all wasted effort.

That rich experience should be the default. If size is a concern, I propose we focus on stripping what users don’t want in a publish phase rather than having options to the detriment of many up front. If someone is concerned about mobile or container sizes, what are the chance we could allow a slimming phase to happen as Xamarin can today?

I don’t see how we can make deployment choices for everyone up at the package level. That’s a deployment concern and a deployment preference. Why is someone’s small footprint deployment priority governing by debugging and error logging experience? The only way to work around this as a package author (as far as I’m aware) is to force the PDB files in the package or to switch to embedded symbols. But both only fulfill me using my own libraries and take the control out of everyone who’s trying to do that small deployment and is okay with sacrificing that functionality. For this reason, I really suggest focus be on splitting deployment concerns into another phase than “how packages are hosted”.

There’s a related issue up there where embedded seems like the best long-term approach, but last we tested .NET Full Framework wouldn’t read the symbols and render the data in stack traces. That’s something that needs fixing if not already done in the .NET 4.x line.

[tmat]

Good discussion. Quite a few points and questions raised here.

First, I’d like to point out that many of these topics are discussed in this document. Some information there is a bit out of date, I’ll update it soon to reflect recent developments.

Let me try to group and answer the questions:

1. Changing the defaults for OSS projects in the .NET Core SDK

I want to propose that the default DebugType in the SDK change from portable to embedded.

We can’t change the default. As @nguerrera mentioned this would be a breaking change. I don’t think we should change the default for v3 projects either. That said …

I think the idea of an opt-in ConfigureOpenSourceDefaults property gives the SDK a chance to provide actionable best practices that people don’t have to think too hard about. That doesn’t change the default, but makes it really easy for developers to follow the guidance for the scenario.

How about creating a package, something like “ProjectBuildPolicies.OpenSource.nupkg” that does what you’re asking for – it sets various properties to values that we deem to be good for OSS projects. No need for setting any property – just referencing the package from a project would opt into OSS settings.

1. Differences between PDBs embedded to DLLs, Portable PDBs included in NuGet packages, Portable PDBs on symbol server, reasoning and guidelines

The difference is in size of DLLs and packages. For some customers and in some scenarios the package size matters, for others it does not. Sometimes the DLL size matters, sometimes it does not.

Why might package size matter? Right now package restore for dotnet/roslyn repository downloads 357 packages of total size 520MB and these take 2.6 GB in NuGet cache. Increasing that means that every restore on a clean machine is slower and needs to move more data around. Why make the restore slower and take more space on disk by downloading symbols for all packages when they will not likely be used? Why should a developer wait longer for restore when they might not even need to debug thru 3rd party code?

Similarly to DLL size. For some trivial utilities the extra size of the .exe/dll doesn’t matter. On the other hand, consider Visual Studio. It may load hundreds of assemblies to devenv.exe process. Since this is a 32-bit process the address space is tight. We can’t afford increasing the size of loaded DLLs unnecessarily. Therefore Embedded PDBs are not an option for anything that ships in VS.

We designed the NuGet symbol acquisition to be pay-for-play. Many customers don’t ever want to step to 3rd party library. There is even a VS debugger option “Just My Code” specifically designed for these customers. The symbols should be available only when they are needed. There are some gaps and inefficiencies in the symbol acquisition tooling (see [5] below). These issues can and should be fixed in such a way that does not regress other scenarios.

As for the guidelines - I don’t think there is a silver bullet approach that works for all cases. It really depends on what the DLLs and PDBs are used for. I’d suggest:

to use Embedded PDBs (DebugType = embedded)

• for simple tools, utilities, build tasks, etc. that are internal to your organization or not expected to be used/referenced often (as in hundreds of thousands)
• when building in CI builds – e.g. Roslyn binaries are built with embedded PDBs for PR validation. When tests crash during CI the crash dump has all symbols in it. We don’t need to publish these symbols anywhere.
• for first party binaries deployed into production – e.g. Bing team builds assemblies that implement Bing services with Embedded PBDs and embedded sources. In their case the significant increase in size is less important than the benefit of having the DLLs completely self-contained, since they have many teams contributing many (thousands) dlls from many repositories. Managing source/symbol servers would be too complex.

to use Portable PDBs packaged in .snupkg

• for all libraries published to NuGet.org

stop adding Portable PDBs to .nuget packages

• this was temporary workaround that was needed before NuGet.org supported symbol server
• if there are scenarios that are not working without PDBs in the package please file issues and let’s get them fixed

1. Enable Source Link by default, or easier to enable

I would also propose that Sourcelink be included/used by default.

For sourcelink, you can add all of them and it’ll use the right one. In fact it’s the recommended approach for dealing with multiple hosts. Sourcelink could be bundled in the SDK like NuGet, and the others.

Unfortunately, this is not possible. In general the user needs to specify what source control provider is hosting the repository. It does so by selecting the right Source Link package (Microsoft.SourceLink.GitHub, Microsoft.SourceLink.GitLab, etc.) For some providers, like GitHub.com or Azure DevOps, it is possible to infer the provider from the remote origin URL since these have well-known domains. However, for on-prem hosted providers (e.g. GitHub Enterprise, GitLab, TFS, etc.) there is no way Source Link can determine the provider from the domain. In these cases a Source Link package referenced by the project checks whether it is the only Source Link package referenced by the project. If so it assumes that the domain belongs to the source control provider it supports. If multiple Source Link packages are referenced then the project needs to specify domains that should be handled by each of them. Besides that, some providers always require a bit of configuration – e.g. projects in TFS need to specify server virtual directory as it is impossible to infer from the URL.

There are improvements I’d like to see: Source Link being easier. Off by default but easy to enable.

What specific improvements do you want to see? Source Link packages are explicitly designed to be as simple to use as possible.

For the most common scenario enabling Source Link is done by referencing the package corresponding to the source control provider that hosts the repository, with no other configuration needed: https://github.com/dotnet/sourcelink#using-sourcelink.

1. Including/excluding symbols when deploying apps

That’s a deployment concern and a deployment preference. Why is someone’s small footprint deployment priority governing by debugging and error logging experience?

maybe we should add a strip task that may run as part of the Release configuration to remove debug symbols

Some things like Appx/MSIX packaging strip-out pdb files by default. They rely on extra properties (AppxPackageIncludePrivateSymbols) to include them,

Sounds like some tools already strip them. I agree that the developer should have the option to include/exclude all symbols when deploying their app. Maybe what is missing is a unified approach (setting) that works across all tools that produce deployment artifacts (Appx, MSIX, .NET Native, IL Linker, etc.)

Some .NET tools, like .NET Native do not support symbol server; the symbols need to be alongside the dll

Seems like something that can and should be fixed. I’d recommend filing a separate issue.

1. Symbol acquisition

Symbol server significantly slows down VS debug startup as it probes for files

Definitely agree with @vancem that this a bug that needs to be fixed and there is no reason to change the design of PDB delivery via NuGet.

Symbol servers may be blocked by corporate firewalls, or otherwise affected by low-bandwidth connections (think airplanes, trains, mobile hotspots, and anywhere else with slower/spotty connections).

Is the firewall blocking symbol servers an issue for symbols.nuget.org? If a firewall is blocking nuget.org then you won’t be able to get packages in the first place.

I agree that symbol acquisition can be improved to accommodate the offline scenario. Today you can run nuget restore to fetch all packages from the server and then work offline. I think we need a similar tool to acquire all symbols you’ll need to debug your project and its dependencies. An msbuild task or a global tool that determines the dependencies and downloads their symbols for specified platform/runtime. Similarly it can take a memory dump file and fetch symbols for all binaries in the dump. Similarly to nuget restore, which uses nuget cache, the tool would populate the local symbol cache debuggers uses to avoid downloading symbols multiple times. This would thus not require any changes in debuggers. They would just consult the cache as they do already. Although, perhaps the debuggers (or even VS and nuget?) might need an easy to use offline mode switch to avoid attempting to download symbols that are not in the cache.

1. Support for Portable PDBs in .NET Framework runtime stack traces

.NET Full Framework wouldn’t read the symbols and render the data in stack traces. That’s something that needs fixing if not already done in the .NET 4.x line.

Portable PDBs (and embedded PDBs) work on 4.7.2. They are not enabled in stack traces by default due to back-compat concerns. A configuration switch is needed.