Support pluggable Settings Backends and Secrets Backends

Related to pluggable STATE backend discussion:

• #5981
• #6270

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/2918

Originally created by @kgpayne on 2021-09-03 09:33:09

---

Problem to solve

Meltano should support multiple secrets backends so that secret config values can be managed in the same way that general config is.

Meltano provides a convenient, centralised configuration management and injection layer for managing all the tools in the data science workflow. However this doesn’t currently include secret config values such as passwords, tokens and keys. This sensitive data must currently be handled by the user outside of meltano, with secrets being injected at runtime using environment variables. This is both an inconvenience for regular users and a barrier to entry for people new to meltano.

Backends to Prioritize

• AWS Parameter Store
• Azure Key Vault
• …

Target audience

• Users deploying meltano in a production environment.
• Users who want to get or set setting values and secrets values from an external authority.

Proposal

Yaml for adding an environment-specific parameter store:

# ...
environments:
  dev:
    settings_backend: <plugin-name>
    secrets_backend: <plugin-name>
    # ...
  staging:
    settings_backend: <plugin-name>
    secrets_backend: <plugin-name>
    # ...
  prod:
    settings_backend: <plugin-name>
    secrets_backend: <plugin-name>
    # ...

Resolving settings at runtime (first match wins):

1. Secrets backend (if set and if kind: password)
2. Settings backend (if set)
3. environment variables
4. .env file (aka dotenv)
5. systemdb
6. yaml

Implementation thoughts

• The secrets backend should be pluggable, to support all the different places secrets can be stored today (AWS Parameter Store, Hashicorp Vault, git-crypt etc.).
• Meltano should supply a default secrets backend, with either file-based encryption (git-crypt) or its database (with password hashing, ala Airflow).

What does success look like, and how can we measure that?

Users can manage all config through meltano. Security best practices are built-in rather than left as an exercise to the user.

Plugin Commands Required

These commands are required to be declared on any settings_backend or secrets_backend plugin:

• list_keys - list all keys with values that have been set
• get_key_value - get a specific key value
• set_key_value - set a specific key value
• clear_key_value - clear a specific key value
• check_key_exists - check if a key exists with a value set

Related but out of scope

• https://github.com/meltano/meltano/issues/3346

Promotion

We’ll want a blog post to go out about this https://github.com/meltano/internal-marketing/issues/530

[pandemicsyn]

Peeking at https://github.com/apache/airflow/blob/main/airflow/secrets/base_secrets.py#L25:7 and a sample implementation for Hashicorp Vault: https://github.com/apache/airflow/blob/main/airflow/providers/hashicorp/secrets/vault.py

Could we get a quick win by just having the first version of a secrets plugin be one that just essentially wrap’s the Airflow secrets backends in a cli?

[aaronsteers]

@cjohnhanson - re:

Do we necessarily want to treat secrets that differently than other settings? One approach to this would be to just implement secrets backends as SettingsValueStores.

I’ve been considering the same and I raised the same today to @tayloramurphy. I’m renaming this issue to reflect the backend as a hybrid parameter/setting/secret backend - similar to AWS Parameter Store’s capability of handling Secrets as well as non-secret config options.