Switch to runtime configuration YAML

As of today, all the runtime configuration is done via environment variables which is fine if you have a few of them but it feels like the list is becoming too long.

We should take a step back and see if we should go to a configuration model where we define them in YAML as well and load them at runtime, similar to how Prometheus and other tools approach this.

Benefits:

• Everything in one file which can be loaded via a volume (Docker) or config map (Kubernetes)
• Configuration can be added to source control
• Easier to read and get an idea what it is configured like
• Centralized configuration

Downsides:

• Harder to get started as you need to know how volumes/config maps work
  • I think we can provide the required guidance around this and/or provide default config, either in the container or docs, to use
• Existing customers need to migrate to the new model

The (de)serialization should use the out-of-the-box capability, and not built our own as we do for the metrics, given it’s pretty straight forward.

Checklist

• Move all configuration to the structure below, except for auth information
• Update all documentation on this and use an example. Location should be /config/runtime.yaml
• Provide unit testing, where needed.
• Update Docker Hub bot.
• Ensure optional variables are working fine
• Update Helm Chart
• Change landing page to point to v1.0

Configuration Example

This configuration should include everything from our configuration docs except for authentication config which would still be via environment variables.

server:
  httpPort: 80
prometheus:
  scrapeEndpoint:
    baseUriPath: /prometheus/scrape
metricsConfiguration:
  absolutePath: /config/metrics-declaration.yaml
logging:
  applicationInsights:
    instrumentationKey: ABC
    isEnabled: true
  containerLogs:
    isEnabled: true
    verbosity: warning
featureFlags:
  metrics:
    disableTimestamps: true # false by default

[jdevalk2]

I like the idea of the config file but consider being able to override every setting with an environment variable as well. Example: How Grafana does things.

Ex.

[server] name=bla

Can be overridden with environment variable GF_SERVER_NAME=bla2

Also consider not making the config file more than 2 levels deep, and prefer 1 level deep, keeping things at the root. I say this because tools like Helm mainly do not allow deep-merge. You can see this in prometheus operator chart where if you want to override a setting 3 levels deep you have to specify the entire subtree from the root downwards.

Also consideration is to be able to mount passwords on filesystem. If there are any passwords in the configuration then ideally those can be mounted through a mounted file secret (instead of over an environment variable). I.e. the secret is mounted on a folder and is contained in a file in that folder. The idea here is that an environment / environment variables are relatively easy to dump when software get compromised and if that dump contains your database credentials that could be used to escalate any attack that was going on.

[ResDiaryLewis]

Hey Tom, this is definitely preferable for us 🙆‍♂. It’s just a bit easier to view the config file than to print out each environment variable, it’s also easier for us to just plug the YAML file in from our Helm values file directly.