Slow step through and watch variable expansion when debugging in a Docker contianer

There’s a performance difference between native debugging and Docker container debugging. The problem can be observed when expanding variables in VS Code. In this test project https://github.com/jamiegs/dotnet-debugging-performance the difference between native and docker is small but still visible. On a real code base, it is much more significant. See this video https://youtu.be/V_jWtOHjvOg for a demonstration.

The details here are on macOS + VS Code. But we have observed the same on Window + VS Code and Windows + VS. Run on high spec Mac Book Pros and Dell XPS hardware. Changes to resources allocated to the container does not seem to make a difference.

Environment data

.NET Core SDK (reflecting any global.json):
 Version:   2.1.300
 Commit:    adab45bf0c

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.13
 OS Platform: Darwin
 RID:         osx.10.13-x64
 Base Path:   /usr/local/share/dotnet/sdk/2.1.300/

Host (useful for support):
  Version: 2.1.2
  Commit:  811c3ce6c0

.NET Core SDKs installed:
  1.0.1 [/usr/local/share/dotnet/sdk]
  1.0.4 [/usr/local/share/dotnet/sdk]
  1.1.4 [/usr/local/share/dotnet/sdk]
  2.0.0 [/usr/local/share/dotnet/sdk]
  2.0.3 [/usr/local/share/dotnet/sdk]
  2.1.4 [/usr/local/share/dotnet/sdk]
  2.1.300-preview2-008533 [/usr/local/share/dotnet/sdk]
  2.1.300 [/usr/local/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.0-preview2-final [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.0-preview2-final [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.1 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.9 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.3 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.0-preview2-26406-04 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

VS Code version:

1.25.1

C# Extension version:

1.15.2

Steps to reproduce

Based on code in https://github.com/jamiegs/dotnet-debugging-performance

There are two Launch Methods, Docker and Native.

• Add breakpoint to ‘return View()’ line in Home Controller. HomeController.csd
• Run application
• When it hits breakpoint, expand the ‘this’ variable. Natively it takes about 3 seconds, Docker takes 5.
• • While this difference isn’t that big, the difference grows dramaticly with larger applications.
• Also hitting ‘Step over’ or ‘Step into’ is slower in Docker vs Native.

Expected behavior

• I would expect docker and native to have very little difference in debugging performance.

Actual behavior

• The difference is noticable and increases with application size.

Demo video comparison using https://github.com/jamiegs/dotnet-debugging-performance https://youtu.be/lkDnFPGOumM

Demo video comparison using a larger, 20,000 line application. https://youtu.be/V_jWtOHjvOg

[ssimek]

Hi Everybody,

I may have found some explanations for the performance issues. Recently I decided to do development in an environment that is as production-like as possible, using the same containers used in production and just docker exec-ing vsdbg inside them for debugging. I immediately hit the issue described here - watches, stepping and the like suddenly took seconds (running on Docker Desktop for Mac).

The first thing I noticed was that during the expression evaluation “hang”, the com.docker.hyperkit process pegs one CPU core, while running top inside the docker VM shows just a few percent for the vsdbg process.

The other one was the following in Console:

ptrace attach of "dotnet <project>.dll"[2451] was attempted by "/vsdbg/vsdbg --interpreter=vscode"[5554]

So it seems like vsdbg is not able to use ptrace at all! Playing around with it some more, I’ve found that the docker VM has kernel.yama.ptrace_scope = 1 set in sysctl, which effectively means that only descendants of a process may be ptrace-d (https://www.kernel.org/doc/Documentation/security/Yama.txt). Changing this to 0 is probably not practical (does not survive a reboot of the docker host VM which is locked down pretty tight with a read-only boot volume), but using docker exec --privileged for the vsdbg process is actually quite enough. After doing this, the above warning disappeared and debugging is (at least subjectively) even faster than regular local debugging on a Mac!

Not sure if my fix is applicable to other scenarios here, but I definitely hope it will help to move this issue forward.

[chuckries]

@janaka @jamiegs One thing that I meant to address is how this relates to Visual Studio vs VS Code. When debugging on docker, both VS and VS Code run the cross platform C# debugger “vsdbg” inside the container. The performance of both frontends should be similar for docker scenarios; they are both dependent on vsdbg inside the container.