Project-level Parameters

Hi, I wondering about the role of parameters and Digdag, using the CLI.

On the one hand, we can set parameters when the server starts as server level parameters shared across projects and workflows using digdag server --param …. On the other hand, it seems we can set workflow level parameters with digdag start … --param ….

What I’m thinking is that I was to be able to set properties for projects and workflows per project, and for the life of the project (not for a single session as, if I understand correctly, with digdag start.

Wondering if support --param makes sense with digdag push … to set project level params, either one at a time or with a config file just like with the server command. That would allow me to change params when changing the project files, or possibly to only change params using digdag push.

Hope this makes sense. I guess my ask then is:

• what is the best way to alter parameters per project?
• is it possible to allow for push to set project level params?

Thanks!

[muga]

@friendofasquid @hiroyuki-sato Sorry for my delayed response. I believe that https://github.com/treasure-data/digdag/pull/845 could solve your issue. https://github.com/treasure-data/digdag/pull/845 enables storing parameter values generated by the current workflow session then, they could be accessed by the next session or other workflows. Please take a look and if you have any questions and suggestions, please feel free to comment on it.

[friendofasquid]

@nickhall I never did get what I needed into production, but the param_get and param_set features look like they’d do what we need. See: https://docs.digdag.io/operators/param_get.html

It’s not 100% obvious how we’d set different values depending on what environment we’re running the code in without restarting the server. One way would be to store that information in a database table and use a digdag operator to query and store. In this case, you’d store your config as a table instead of as a YAML file.