feat: add support for `env` key for `jobs`
See original GitHub issueSee discussion in slack: https://meltano.slack.com/archives/C01TCRBBJD7/p1657233220239139
Consider example:
jobs:
- name: gitlab
tasks:
- dbt-postgres:seed tap-gitlab target-postgres dbt-postgres:snapshot dbt-postgres:run dbt-postgres:test
meltano run gitlab
will run all the dbt commands for the whole project rather than allowing me to specify only the specific models that have changed (e.g. DBT_MODELS=+gitlab+
).
I realize this can be solved by schedules:
schedules:
- name: gitlab
interval: '@daily'
job: gitlab
env:
DBT_MODELS: +gitlab+
However it seems to me that this information belongs to the job itself, not the schedule; that is even if I were to manually run meltano run gitlab
outside of the @daily
schedule, I’d still expect it to apply my configuration.
Proposal: Allow env
s to be set on a job basis:
jobs:
- name: gitlab
tasks:
- dbt-postgres:seed tap-gitlab target-postgres dbt-postgres:snapshot dbt-postgres:run dbt-postgres:test
env:
DBT_MODELS: +gitlab+
TARGET_POSTGRES_BATCH_SIZE_ROWS: 100 # could also be useful to override plugin configs
Precedence can get tricky here, but I would expect:
meltano environment < job < schedule < .env < system envrionment
Issue Analytics
- State:
- Created a year ago
- Comments:8 (3 by maintainers)
Top Results From Across the Web
Workflow level env does not work properly in all fields. #480
When using workflow level environment variables they do not work when I use them to substitute for jobs.build.name or jobs.build.runs-on .
Read more >EnvKey - Secure configuration management.
End-to-end encryption for app secrets. Superpowers for managing config. No more sloppy secrets sharing → protect API keys, encryption keys, and credentials.
Read more >Configuring Environment Variables and Secrets - Render
Click on Environment in the left pane, and then click on Add Environment Variable. Service environment. Enter the Key and Value for your...
Read more >Environment Variables in Next.js - Frontend Digest
This creates an environment variable called GREETING which contains the string Hello world . Note that you need to restart Next.js to apply...
Read more >Accenture | Let there be change
Using technology and human ingenuity to meet environmental goals for a powerful, sustainable future. explore. Careers Blog. My career path to digital ...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
@aaronsteers @cjohnhanson I’m pulling this out of the next iteration. While I think this would be a useful thing to have, I don’t think we need to currently build it ourselves. Thinking through the spec discussion has been useful and clarifying in relation to #6387 but adding support for
env
at the jobs level isn’t critical right now.I’ll mark it as accepting PR’s though in case the community is interested in having this.
@cjohnhanson I think this would be the best order:
My logic here is similar to what I said in https://github.com/meltano/meltano/issues/6386#issuecomment-1179138398 But basically, a job is a collection of plugins run at the same time and users would likely want to override plugin settings at the job level. Similarly, a schedule is a particular instance of a job and there can be multiple schedules per job, so overrides from there make sense above jobs. Then having the environment level plugin env as the highest priority is the last place you could override anything set lower in the hierarchy.
@aaronsteers would you agree on this?