Allow users to pass credentials through environment variables

Description

In the new config loader (https://github.com/kedro-org/kedro/issues/1868), allow users to pass credentials through environment variables using the OmegaConf resolver oc.env: https://omegaconf.readthedocs.io/en/latest/custom_resolvers.html#oc-env

Context

The “default” way for Kedro to deal with credentials is through the credentials.yml file. @AntonyMilneQB has explained very thoroughly how this works and why it’s problematic here: https://github.com/kedro-org/kedro/issues/1646

Over time many users have flagged that they want to be able to pass credentials through environment variables. Currently, this is possible if users use the TemplatedConfigLoader, but ideally this functionality would be available by default.

Insight 11 of the configuration research also showed that users are currently storing credentials in environment variables: https://github.com/kedro-org/kedro/issues/1847

Implementation

• The new configloader will use OmegaConf instead of anyconf https://github.com/kedro-org/kedro/issues/1868

• The built-in resolvers will be turned off

• In this task add logic to load credentials using OmegaConf with the oc.env resolver turned on. This might mean separating the credentials loading logic so that the resolver for that loading process hasn’t been turned off. If this is impossible or really difficult, we’ll allow env variables for all config, but that should be discussed.

• Document how should users pass credentials - as current way to do it is via `TemplatedConfigLoader

[szczeles]

@MerelTheisenQB Thanks for asking! I’ve added 2 more comments with challenges that we faced with the current configuration parsing logic during Kedro deployments:

• inability to pass complex data types as --params
• configs merging behaviour idea

[szczeles]

@MerelTheisenQB That’s a great idea to move to the config framework that supports env variables! In kedro-kubeflow plugin we provide custom EnvTemplatedConfigLoader, for example, to fulfill a requirement of adjusting the docker image name to the current commit when running the pipeline in Kubeflow Pipelines environment (see https://github.com/getindata/kedro-kubeflow/blob/0.7.3/kedro_kubeflow/context_helper.py#L13)

Having this solved out-of-the-box by OmegaConf looks really neat!