Dump-Creation crashes with FileNotFoundException

Description

When I trigger a dump (via the /dump Endpoint) with the setup described below, the request fails with a FileNotFoundException: The exception occurs, when the dumps is created and should be served as HTTP-Response.

{"Timestamp":"2021-08-03T09:08:35.8444255Z","EventId":50,"LogLevel":"Warning","Category":"Microsoft.AspNetCore.DataProtection.Repositories.EphemeralXmlRepository","Message":"Using an in-memory repository. Keys will not be persisted to storage.","State":{"Message":"Using an in-memory repository. Keys will not be persisted to storage.","{OriginalFormat}":"Using an in-memory repository. Keys will not be persisted to storage."},"Scopes":[]}
{"Timestamp":"2021-08-03T09:08:35.8569902Z","EventId":59,"LogLevel":"Warning","Category":"Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager","Message":"Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits.","State":{"Message":"Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits.","{OriginalFormat}":"Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits."},"Scopes":[]}
{"Timestamp":"2021-08-03T09:08:35.9054231Z","EventId":35,"LogLevel":"Warning","Category":"Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager","Message":"No XML encryptor configured. Key {2ef34f2a-9ba8-49a3-a7b8-3dc88af4e8a1} may be persisted to storage in unencrypted form.","State":{"Message":"No XML encryptor configured. Key {2ef34f2a-9ba8-49a3-a7b8-3dc88af4e8a1} may be persisted to storage in unencrypted form.","KeyId":"2ef34f2a-9ba8-49a3-a7b8-3dc88af4e8a1","{OriginalFormat}":"No XML encryptor configured. Key {KeyId:B} may be persisted to storage in unencrypted form."},"Scopes":[]}
{"Timestamp":"2021-08-03T09:08:35.9643413Z","EventId":14,"LogLevel":"Warning","Category":"Microsoft.Diagnostics.Tools.Monitor.Startup","Message":"WARNING: Authentication is enabled over insecure http transport. This can pose a security risk and is not intended for production environments.","State":{"Message":"WARNING: Authentication is enabled over insecure http transport. This can pose a security risk and is not intended for production environments.","{OriginalFormat}":"WARNING: Authentication is enabled over insecure http transport. This can pose a security risk and is not intended for production environments."},"Scopes":[]}
{"Timestamp":"2021-08-03T09:08:36.0514869Z","EventId":0,"LogLevel":"Warning","Category":"Microsoft.AspNetCore.Server.Kestrel","Message":"Overriding address(es) \u0027http://*:52323\u0027. Binding to endpoints defined in UseKestrel() instead.","State":{"Message":"Overriding address(es) \u0027http://*:52323\u0027. Binding to endpoints defined in UseKestrel() instead.","addresses":"http://*:52323","methodName":"UseKestrel()","{OriginalFormat}":"Overriding address(es) \u0027{addresses}\u0027. Binding to endpoints defined in {methodName} instead."},"Scopes":[]}
{"Timestamp":"2021-08-03T09:08:36.0646374Z","EventId":0,"LogLevel":"Information","Category":"Microsoft.Hosting.Lifetime","Message":"Now listening on: http://[::]:52323","State":{"Message":"Now listening on: http://[::]:52323","address":"http://[::]:52323","{OriginalFormat}":"Now listening on: {address}"},"Scopes":[]}
{"Timestamp":"2021-08-03T09:08:36.0660690Z","EventId":0,"LogLevel":"Information","Category":"Microsoft.Hosting.Lifetime","Message":"Application started. Press Ctrl\u002BC to shut down.","State":{"Message":"Application started. Press Ctrl\u002BC to shut down.","{OriginalFormat}":"Application started. Press Ctrl\u002BC to shut down."},"Scopes":[]}
{"Timestamp":"2021-08-03T09:08:36.0661498Z","EventId":0,"LogLevel":"Information","Category":"Microsoft.Hosting.Lifetime","Message":"Hosting environment: Production","State":{"Message":"Hosting environment: Production","envName":"Production","{OriginalFormat}":"Hosting environment: {envName}"},"Scopes":[]}
{"Timestamp":"2021-08-03T09:08:36.0661920Z","EventId":0,"LogLevel":"Information","Category":"Microsoft.Hosting.Lifetime","Message":"Content root path: /app/.store/dotnet-monitor/5.0.0-preview.6.21370.3/dotnet-monitor/5.0.0-preview.6.21370.3/tools/netcoreapp3.1/any/","State":{"Message":"Content root path: /app/.store/dotnet-monitor/5.0.0-preview.6.21370.3/dotnet-monitor/5.0.0-preview.6.21370.3/tools/netcoreapp3.1/any/","contentRoot":"/app/.store/dotnet-monitor/5.0.0-preview.6.21370.3/dotnet-monitor/5.0.0-preview.6.21370.3/tools/netcoreapp3.1/any/","{OriginalFormat}":"Content root path: {contentRoot}"},"Scopes":[]}
{"Timestamp":"2021-08-03T09:09:22.9267934Z","EventId":5,"LogLevel":"Information","Category":"Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController","Message":"Written to HTTP stream.","State":{"Message":"Written to HTTP stream.","{OriginalFormat}":"Written to HTTP stream."},"Scopes":[{"Message":"ConnectionId:0HMAMBPH6M1OP","ConnectionId":"0HMAMBPH6M1OP"},{"Message":"RequestPath:/info RequestId:0HMAMBPH6M1OP:00000001, SpanId:|ad4f683-467d6a1aa95995a8., TraceId:ad4f683-467d6a1aa95995a8, ParentId:","RequestId":"0HMAMBPH6M1OP:00000001","RequestPath":"/info","SpanId":"|ad4f683-467d6a1aa95995a8.","TraceId":"ad4f683-467d6a1aa95995a8","ParentId":""},{"Message":"Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.GetInfo (Microsoft.Diagnostics.Monitoring.WebApi)","ActionId":"b1fdaee0-707b-4c3c-88d9-76f8484b98c6","ActionName":"Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.GetInfo (Microsoft.Diagnostics.Monitoring.WebApi)"}]}
{"Timestamp":"2021-08-03T09:13:13.6798522Z","EventId":13,"LogLevel":"Error","Category":"Microsoft.AspNetCore.Server.Kestrel","Message":"Connection id \u00220HMAMBPH6M1OP\u0022, Request id \u00220HMAMBPH6M1OP:00000002\u0022: An unhandled exception was thrown by the application.","Exception":"System.IO.FileNotFoundException: Could not find file \u0027/tmp/a70c3228-f0c3-41cc-af93-a0761e4d223d_1\u0027. File name: \u0027/tmp/a70c3228-f0c3-41cc-af93-a0761e4d223d_1\u0027    at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func\u00602 errorRewriter)    at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)    at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)    at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)    at Microsoft.Diagnostics.Monitoring.WebApi.DiagnosticServices.AutoDeleteFileStream..ctor(String path) in /_/src/Microsoft.Diagnostics.Monitoring.WebApi/DiagnosticServices.cs:line 180    at Microsoft.Diagnostics.Monitoring.WebApi.DiagnosticServices.GetDump(IProcessInfo pi, DumpType mode, CancellationToken token) in /_/src/Microsoft.Diagnostics.Monitoring.WebApi/DiagnosticServices.cs:line 104    at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.\u003C\u003Ec__DisplayClass16_0.\u003C\u003CCaptureDump\u003Eb__0\u003Ed.MoveNext() in /_/src/Microsoft.Diagnostics.Monitoring.WebApi/Controllers/DiagController.cs:line 219 --- End of stack trace from previous location where exception was thrown ---    at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.\u003C\u003Ec__DisplayClass36_0.\u003C\u003CInvokeForProcess\u003Eb__0\u003Ed.MoveNext() in /_/src/Microsoft.Diagnostics.Monitoring.WebApi/Controllers/DiagController.cs:line 824 --- End of stack trace from previous location where exception was thrown ---    at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.\u003C\u003Ec__DisplayClass38_0\u00601.\u003C\u003CInvokeForProcess\u003Eb__0\u003Ed.MoveNext() in /_/src/Microsoft.Diagnostics.Monitoring.WebApi/Controllers/DiagController.cs:line 856 --- End of stack trace from previous location where exception was thrown ---    at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagControllerExtensions.InvokeService[T](ControllerBase controller, Func\u00601 serviceCall, ILogger logger) in /_/src/Microsoft.Diagnostics.Monitoring.WebApi/Controllers/DiagControllerExtensions.cs:line 57    at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.InvokeForProcess[T](Func\u00602 func, Nullable\u00601 processKey, String artifactType) in /_/src/Microsoft.Diagnostics.Monitoring.WebApi/Controllers/DiagController.cs:line 846    at Microsoft.Diagnostics.Monitoring.WebApi.Controllers.DiagController.InvokeForProcess(Func\u00602 func, Nullable\u00601 processKey, String artifactType) in /_/src/Microsoft.Diagnostics.Monitoring.WebApi/Controllers/DiagController.cs:line 824    at lambda_method(Closure , Object )    at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()    at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeActionMethodAsync\u003Eg__Awaited|12_0(ControllerActionInvoker invoker, ValueTask\u00601 actionResultValueTask)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeNextActionFilterAsync\u003Eg__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State\u0026 next, Scope\u0026 scope, Object\u0026 state, Boolean\u0026 isCompleted)    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeInnerFilterAsync\u003Eg__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeFilterPipelineAsync\u003Eg__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeAsync\u003Eg__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)    at Microsoft.AspNetCore.Routing.EndpointMiddleware.\u003CInvoke\u003Eg__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)    at Microsoft.Diagnostics.Tools.Monitor.RequestLimitMiddleware.Invoke(HttpContext context) in /_/src/Tools/dotnet-monitor/RequestLimitMiddleware.cs:line 67    at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)    at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)    at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)    at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication\u00601 application)","State":{"Message":"Connection id \u00220HMAMBPH6M1OP\u0022, Request id \u00220HMAMBPH6M1OP:00000002\u0022: An unhandled exception was thrown by the application.","ConnectionId":"0HMAMBPH6M1OP","TraceIdentifier":"0HMAMBPH6M1OP:00000002","{OriginalFormat}":"Connection id \u0022{ConnectionId}\u0022, Request id \u0022{TraceIdentifier}\u0022: An unhandled exception was thrown by the application."},"Scopes":[{"Message":"ConnectionId:0HMAMBPH6M1OP","ConnectionId":"0HMAMBPH6M1OP"},{"Message":"RequestPath:/dump RequestId:0HMAMBPH6M1OP:00000002, SpanId:|ad4f684-467d6a1aa95995a8., TraceId:ad4f684-467d6a1aa95995a8, ParentId:","RequestId":"0HMAMBPH6M1OP:00000002","RequestPath":"/dump","SpanId":"|ad4f684-467d6a1aa95995a8.","TraceId":"ad4f684-467d6a1aa95995a8","ParentId":""}]}

Configuration

ASP.NET Core version: 3.1.17 (based on mcr.microsoft.com/dotnet/core/aspnet:3.1-alpine) Dotnet Monitor: 5.0.0-preview.6.21370.3+9c1714943e349b437ca3adeb8f4f8cb0df1355b3 OS: Alpine (both application and monitor) Arch: x64 Env: Bare-Metal Kubernetes-Cluster (no cloud-provider)

Kubernetes Deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: feature-diagnostics-api
spec:
  selector:
    matchLabels: {}
  template:
    spec:
      containers:
      - image: our.aspnet-core-application:latest
        imagePullPolicy: IfNotPresent
        name: api
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources:
          limits:
            cpu: 1500m
            memory: 1Gi
          requests:
            cpu: 250m
            memory: 400Mi
        securityContext:
          capabilities:
            drop:
            - ALL
          privileged: false
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: diagnostics
      - args:
        - --urls
        - https://*:52323
        env:
        - name: DotnetMonitor_Metrics__Enabled
          value: "false"
        - name: DotnetMonitor_Kestrel__Certificates__Default__Password
          valueFrom:
            secretKeyRef:
              key: pass
              name: monitorcert
        - name: DotnetMonitor_Kestrel__Certificates__Default__Path
          value: /etc/aspnet/customcert/cert.pfx
        - name: DotnetMonitor_ApiAuthentication__ApiKeyHashType
          value: SHA256
        - name: DotnetMonitor_ApiAuthentication__ApiKeyHash
          valueFrom:
            secretKeyRef:
              key: ApiAuthentication__ApiKeyHash
              name: monitorkey
        image: mcr.microsoft.com/dotnet/monitor:5.0.0-preview.6-alpine
        imagePullPolicy: IfNotPresent
        name: monitor-sidecar
        ports:
        - containerPort: 52323
          name: http
          protocol: TCP
        securityContext:
          capabilities:
            drop:
            - ALL
          privileged: false
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: diagnostics
        - mountPath: /etc/aspnet/customcert
          name: customcert
      volumes:
      - emptyDir: {}
        name: diagnostics
      - name: customcert
        secret:
          defaultMode: 420
          secretName: monitorcert

Regression?

Not tested.

Other information

Nothing, please ask if some info is missing.

[mikem8361]

This is probably a dump creation issue on the runtime side in alpine 3.1. dotnet-dump sends a message to the runtime/app process and it exec’s createdump which uses the DAC (libmscordaccore.so) to actually generate the core dump. It is most likely in the DAC because it isn’t used for full dumps. These issues may be fixed in 5.0 and the upcoming 6.0 releases, but the only currently work around for 3.1 is to always generate full dumps.

[gabbersepp]

@wiktork Hi I am a colleague of Christian who is currently in vacation. So I started to investigate this bug further. I have not tested yet your idea of wrong permissions, I will do this as a next step. But from what I see I do not think that it is about a permission issue. When running the dump the /tmp folder does not contain a dump. Also after dotnet-monitor tries to get the dump (and throws the fileNotFound Exception) there is still no dump in the folder. But in both container I see the same socket file of the .net diagnostics port. So the /tmp folder can be accessed by both containers.

My next steps will be:

• try the idea about permission problems
• try to get traces of dotnet-monitor and our application by using dotnet-trace manually. I think there should be Fileevents that may indicate what is wrong

//Edit: Your idea of using another folder did not work as you already expected.