Runtime metrics not generated

Description

When I try to use the dotnet-monitor as a sidecar container, the runtime metrics is not generated

I could only get the following metrics:

It seemed only have microsoftaspnetcorehosting related provider? Not sure if I’m wrong with some config

Configuration

deployment yaml https://github.com/WeihanLi/SparkTodo/blob/82ba4ac7493afcb476d096cf11f482e9a297003a/sparktodo-api-k8s-deploy.yaml#L20

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sparktodo-api
  labels:
    app: sparktodo-api
spec:
  replicas: 1
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      app: sparktodo-api
  minReadySeconds: 0
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1

  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "52323"
      labels:
        app: sparktodo-api
    
    spec:
      containers:
        - name: sparktodo-api
          image: weihanli/sparktodo-api:latest
          imagePullPolicy: Always
          resources:
            requests:
              memory: "64Mi"
              cpu: "20m"
            limits:
              memory: "128Mi"
              cpu: "50m"
          env:
          - name: ASPNETCORE_URLS
            value: http://+:80
          - name: DOTNET_DiagnosticPorts
            value: /diag/port
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /health
              port: 80
            initialDelaySeconds: 60
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /health
              port: 80
            initialDelaySeconds: 60
            periodSeconds: 30
          volumeMounts:
          - mountPath: /diag
            name: diagvol
          - mountPath: /dumps
            name: dumpsvol
        - name: monitor
          image: mcr.microsoft.com/dotnet/monitor:6.0.0
          # DO NOT use the --no-auth argument for deployments in production
          args: [ "--no-auth" ]
          imagePullPolicy: Always
          ports:
            - containerPort: 52323
          env:
          - name: DOTNETMONITOR_DiagnosticPort__ConnectionMode
            value: Listen
          - name: DOTNETMONITOR_DiagnosticPort__EndpointName
            value: /diag/port
          - name: DOTNETMONITOR_Storage__DumpTempFolder
            value: /dumps
          - name: DOTNETMONITOR_Urls
            value: "http://*:52323"
          volumeMounts:
          - mountPath: /diag
            name: diagvol
          - mountPath: /dumps
            name: dumpsvol
          resources:
            requests:
              cpu: 50m
              memory: 32Mi
            limits:
              cpu: 100m
              memory: 256Mi
          securityContext:
            capabilities:
              add: ["SYS_PTRACE"]
      volumes:
      - name: diagvol
        emptyDir: {}
      - name: dumpsvol
        emptyDir: {}

[johan-bjerling]

I’m experiencing the same issue when I run dotnet-monitor as a sidecar in AKS.

I’m able add other providers, e.g. System.Net.Http and Microsoft-AspNetCore-Server-Kestrel and I get metrics for them. I even tried to add System.Runtime manually, both with IncludeDefaultProviders set to true and to false, but to no avail.

[jander-msft]

Although, when I curl http://localhost:52325/metrics I do see metrics getting generated for microsoftaspnetcorehosting, systemruntime and microsoftaspnetcoreserverkestrel but I don’t see them in Prometheus.

If you see systemruntime from .NET Monitor, then it is unlikely that OpenTelemetry is interfering with the collection of metrics by .NET Monitor.

I am assuming it has to do with interval mismatch with OpenTelemetry and .Net Monitor? Is the fix to make sure the interval values match for .Net Monitor and OpenTelemetry? Is there any other change that needs to be made?

.NET Monitor doesn’t automatically provide information to a Promethues server; you have to configure that server to scrape the /metrics route.

I’m not certain that the interval mismatch is impacting anything within the consumption of the Prometheus metrics. @wiktork any ideas?