Runtime config metrics sinks configuration seem broken

My goal is to configure promitor to only report metrics via its own /metrics scraping endpoint for prometheus. I can see in the docs that this configuratio has been changed in version 1.6 (which I am using). However, I cannot get this to work.

Expected Behavior

I can configure Promitor to only use the built-in prometheus scraping endpoint metric sink.

Actual Behavior

I tried 3 different configurations:

1. Use only the new “metrics sinks” configuration:

server:
  httpPort: 80
metricSinks:
  prometheusScrapingEndpoint:
    metricUnavailableValue: NaN
    enableMetricTimestamps: true
    baseUriPath: /metrics
metricsConfiguration:
  absolutePath: /config/metrics-declaration.yaml
telemetry:
  applicationInsights:
    isEnabled: false
    verbosity: trace
  containerLogs:
    isEnabled: true
  defaultVerbosity: "Error"

with this configuration, I get an exception on startup:

██████╗ ██████╗  ██████╗ ███╗   ███╗██╗████████╗ ██████╗ ██████╗
██╔══██╗██╔══██╗██╔═══██╗████╗ ████║██║╚══██╔══╝██╔═══██╗██╔══██╗
██████╔╝██████╔╝██║   ██║██╔████╔██║██║   ██║   ██║   ██║██████╔╝
██╔═══╝ ██╔══██╗██║   ██║██║╚██╔╝██║██║   ██║   ██║   ██║██╔══██╗
██║     ██║  ██║╚██████╔╝██║ ╚═╝ ██║██║   ██║   ╚██████╔╝██║  ██║
╚═╝     ╚═╝  ╚═╝ ╚═════╝ ╚═╝     ╚═╝╚═╝   ╚═╝    ╚═════╝ ╚═╝  ╚═╝
[05:48:50 FTL] Host terminated unexpectedly
System.NullReferenceException: Object reference not set to an instance of an object.
   at Promitor.Agents.Scraper.Startup..ctor(IConfiguration configuration) in /src/Promitor.Agents.Scraper/Startup.cs:line 33
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
   at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass12_0.<UseStartup>b__0(HostBuilderContext context, IServiceCollection services)
   at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Promitor.Agents.Scraper.Program.Main(String[] args) in /src/Promitor.Agents.Scraper/Program.cs:line 23

1. Use only the deprecated configuration

server:
  httpPort: 80
prometheus:
  metricUnavailableValue: NaN # Optional. Default: NaN
  enableMetricTimestamps: false # Optional. Default: true
  scrapeEndpoint:
    baseUriPath: /metrics # Optional. Default: /metrics
metricsConfiguration:
  absolutePath: /config/metrics-declaration.yaml
telemetry:
  applicationInsights:
    isEnabled: false
    verbosity: trace
  containerLogs:
    isEnabled: true
  defaultVerbosity: "Error"

With this, I get a NullReferenceException on startup which seems to indicate (when looking at the source code) that the metrics sinks configuration secgtion is missing (which it is indeed):

██████╗ ██████╗  ██████╗ ███╗   ███╗██╗████████╗ ██████╗ ██████╗ 
██╔══██╗██╔══██╗██╔═══██╗████╗ ████║██║╚══██╔══╝██╔═══██╗██╔══██╗
██████╔╝██████╔╝██║   ██║██╔████╔██║██║   ██║   ██║   ██║██████╔╝
██║     ██║  ██║╚██████╔╝██║ ╚═╝ ██║██║   ██║   ╚██████╔╝██║  ██║
╚═╝     ╚═╝  ╚═╝ ╚═════╝ ╚═╝     ╚═╝╚═╝   ╚═╝    ╚═════╝ ╚═╝  ╚═╝
[05:51:33 FTL] Host terminated unexpectedly
System.NullReferenceException: Object reference not set to an instance of an object.
   at Promitor.Agents.Scraper.Startup.BuildOpenApiDescription(IConfiguration configuration) in /src/Promitor.Agents.Scraper/Startup.cs:line 111
   at Promitor.Agents.Scraper.Startup.ConfigureServices(IServiceCollection services) in /src/Promitor.Agents.Scraper/Startup.cs:line 39
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass12_0.<UseStartup>b__0(HostBuilderContext context, IServiceCollection services)
   at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Promitor.Agents.Scraper.Program.Main(String[] args) in /src/Promitor.Agents.Scraper/Program.cs:line 23

1. Use both the deprecated and the new configuration

server:
  httpPort: 80
metricSinks:
  prometheusScrapingEndpoint:
    metricUnavailableValue: NaN
    enableMetricTimestamps: true
    baseUriPath: /metrics
prometheus:
  metricUnavailableValue: NaN # Optional. Default: NaN
  enableMetricTimestamps: false # Optional. Default: true
  scrapeEndpoint:
    baseUriPath: /metrics # Optional. Default: /metrics
metricsConfiguration:
  absolutePath: /config/metrics-declaration.yaml
telemetry:
  applicationInsights:
    isEnabled: false
    verbosity: trace
  containerLogs:
    isEnabled: true
  defaultVerbosity: "Error"

With this, I get a validation error indicating that I specified duplicate metric sinks, which I did indeed:

[05:53:55 INF] Validation step 3/5 succeeded
[05:53:55 INF] Start Validation step 4/5: StatsD Metric Sink
[05:53:55 INF] Validation step 4/5 succeeded
[05:53:55 INF] Start Validation step 5/5: Prometheus Scraping Endpoint Metric Sink
[05:53:55 WRN] Validation step 5/5 failed. Error(s): Duplicate Prometheus scraping endpoint was configured in the metric sink & legacy configuration
[05:53:55 FTL] Promitor is not configured correctly. Please fix validation issues and re-run.
[05:53:55 FTL] Host terminated unexpectedly
Promitor.Agents.Scraper.Validation.Exceptions.ValidationFailedException: Validation Failed. Errors:- Prometheus Scraping Endpoint Metric Sink: Duplicate Prometheus scraping endpoint was configured in the metric sink & legacy configuration

   at Promitor.Agents.Scraper.Validation.RuntimeValidator.ProcessValidationResults(List`1 validationResults) in /src/Promitor.Agents.Scraper/RuntimeValidator.cs:line 60   at Promitor.Agents.Scraper.Validation.RuntimeValidator.Run() in /src/Promitor.Agents.Scraper/RuntimeValidator.cs:line 47
   at Promitor.Agents.Scraper.Startup.ValidateRuntimeConfiguration(IServiceCollection services) in /src/Promitor.Agents.Scraper/Startup.cs:line 86
   at Promitor.Agents.Scraper.Startup.ConfigureServices(IServiceCollection services) in /src/Promitor.Agents.Scraper/Startup.cs:line 51
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass12_0.<UseStartup>b__0(HostBuilderContext context, IServiceCollection services)
   at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Promitor.Agents.Scraper.Program.Main(String[] args) in /src/Promitor.Agents.Scraper/Program.cs:line 23

The only way I got this working at all was to specify the prometheus scraping endpoint using the legacy configuration, and specify ANOTHER sink (statsd) using the new metricsSinks configuration.

Steps to Reproduce the Problem

See actual behaviour.

Specifications

• Version: 1.6.0
• Platform: Windows 10 Host, using Linux Containers

[tomkerkhove]

Yes, I’m fixing that and shipping 1.6.1 since there is a default value messing things up which I didn’t use in testing - Sorry!

[ChristianEder]

that one fixes it, yes