Allow defining credentials from variable environment

Description

It is not clear how one can best deploy Kedro with credentials for production. A good practice in traditional applications is to put sensitive credentials using environment variables.

In Kedro, it looks like it relies only on credentials.yml, located somewhere except in conf/local/ folder.

(PS: If I’m missing something with this, the problem is probably bad or missing documentation since I’m not able to find this information in the doc)

Context

Using environment variables will help standardize the deployment of Kedro like any app, thus reduces learning curve for developers.

Possible Implementation

Possible change could be to look for environment variables first, before looking for the content of credentials.yml. Changes will mainly be located in ConfigLoader class.

Possible Alternatives

I create credentials.yml file in config/base/ folder, because this folder ignored by Git, but is still packaged in the Kedro Docker. This is still not good because the credentials are now located in the Docker images repository: Anyone can pull that image and get prod credentials!

---

If this is an interesting implementation (which I think it is), I would be happy to contribute by implementing it. I first want to open the discussion about the best implementation given Kedro orientation, before making any pull request.

Regards,

[DmitriiDeriabinQB]

@htaidirt, thank you for your contribution. Please find some comments below.

It is not clear how one can best deploy Kedro with credentials for production.

Currently Kedro supports deploying credentials for production via configuration environments. Default project template contains 2 of them - conf/base is intended for storage of non-sensitive shareable configuration, and conf/local, where you can put sensitive credentials. conf/local is in .gitignore by default. Please find more information on how kedro configuration module works in this section of documentation.

I create credentials.yml file in config/base/ folder, because this folder ignored by Git, but is still packaged in the Kedro Docker. This is still not good because the credentials are now located in the Docker images repository: Anyone can pull that image and get prod credentials!

conf/base is indeed copied into Docker image, however, as documentation suggests, it is not intended to store any credentials. You should rather store your credentials in conf/local/credentials.yml, which is in .dockerignore by default.

Using environment variables will help standardize the deployment of Kedro like any app, thus reduces learning curve for developers.

Currently you can manually construct/enrich your credentials dictionary in src/<package_name>/run.py with any data, including one coming from the environment variables.

In long term we consider adding templating capability for kedro configs, which may, possibly, handle environment variables, however exact specification hasn’t been finalised yet.

[HugoPerrier]

I posted my solution in How do I fill the credentials from environment variables #403

Thanks for the help