Question: how to access credentials inside a node?

What are you trying to do?

I am trying to perform ETL operations inside a node. To do this, I need access to database credentials.

Workaround

As a workaround, I can add sqlalchemy engine to DataCatalog in register_data_catalog hook dynamically, but I don’t think DataCatalog should be used in this way.

        catalog.add_feed_dict(
            {"mssql_engine": MemoryDataSet(self._mssql_engine, copy_mode='assign')}
        )

[atgmello]

I have a similar usecase as the one mentioned by @bensdm. I also could not find a built-in way of getting this data from the credentials.yml. For now I simply wrap my credential loading in a node and pass the resulting dict as an input to wherever they’re needed. Something along the lines of:

# credentials.yml
# (...)
app:
    client_id: abc
    client_secret: xyz

# nodes.py
import yaml
# (...)
def get_app_credentials():
    with open("./conf/local/credentials.yml") as cred:
        cred_dict = yaml.safe_load(cred).get("app")
    return cred_dict

def authenticate_user(credentials):
  # (...)
  return something

# pipeline.py
# (...)
def create_pipeline(**kwargs):
    return Pipeline(
        [
            node(
                func=get_app_credentials,
                inputs=None,
                outputs="credentials"
            ),
            node(
                func=authenticate_user,
                inputs="credentials",
                outputs="something",
            ),
            # (...)
        ]
    )

Any thoughts or comments on whether this is an appropriate work-around or not are very much welcome!

[bensdm]

interested by the answer too, what if i need credentials to use let’s say a google API inside a node? how can I pass it as a param?