question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

feat: add support for `env` key for `jobs`

See original GitHub issue

See discussion in slack: https://meltano.slack.com/archives/C01TCRBBJD7/p1657233220239139

Consider example:

jobs:
- name: gitlab
  tasks:
  - dbt-postgres:seed tap-gitlab target-postgres dbt-postgres:snapshot dbt-postgres:run dbt-postgres:test

meltano run gitlab will run all the dbt commands for the whole project rather than allowing me to specify only the specific models that have changed (e.g. DBT_MODELS=+gitlab+).

I realize this can be solved by schedules:

schedules:
- name: gitlab
  interval: '@daily'
  job: gitlab
  env:
    DBT_MODELS: +gitlab+

However it seems to me that this information belongs to the job itself, not the schedule; that is even if I were to manually run meltano run gitlab outside of the @daily schedule, I’d still expect it to apply my configuration.

Proposal: Allow envs to be set on a job basis:

jobs:
- name: gitlab
  tasks:
  - dbt-postgres:seed tap-gitlab target-postgres dbt-postgres:snapshot dbt-postgres:run dbt-postgres:test
  env:
    DBT_MODELS: +gitlab+
    TARGET_POSTGRES_BATCH_SIZE_ROWS: 100 # could also be useful to override plugin configs

Precedence can get tricky here, but I would expect:

meltano environment < job < schedule < .env < system envrionment

Issue Analytics

  • State:open
  • Created a year ago
  • Comments:8 (3 by maintainers)

github_iconTop GitHub Comments

2reactions
tayloramurphycommented, Aug 2, 2022

@aaronsteers @cjohnhanson I’m pulling this out of the next iteration. While I think this would be a useful thing to have, I don’t think we need to currently build it ourselves. Thinking through the spec discussion has been useful and clarifying in relation to #6387 but adding support for env at the jobs level isn’t critical right now.

I’ll mark it as accepting PR’s though in case the community is interested in having this.

1reaction
tayloramurphycommented, Jul 26, 2022

@cjohnhanson I think this would be the best order:

- environment level plugin env # highest
- schedule level env
- job level env
- root level plugin env 
- environment level env 
- root level env
- .env file
- terminal env # lowest

My logic here is similar to what I said in https://github.com/meltano/meltano/issues/6386#issuecomment-1179138398 But basically, a job is a collection of plugins run at the same time and users would likely want to override plugin settings at the job level. Similarly, a schedule is a particular instance of a job and there can be multiple schedules per job, so overrides from there make sense above jobs. Then having the environment level plugin env as the highest priority is the last place you could override anything set lower in the hierarchy.

@aaronsteers would you agree on this?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Workflow level env does not work properly in all fields. #480
When using workflow level environment variables they do not work when I use them to substitute for jobs.build.name or jobs.build.runs-on .
Read more >
EnvKey - Secure configuration management.
End-to-end encryption for app secrets. Superpowers for managing config. No more sloppy secrets sharing → protect API keys, encryption keys, and credentials.
Read more >
Configuring Environment Variables and Secrets - Render
Click on Environment in the left pane, and then click on Add Environment Variable. Service environment. Enter the Key and Value for your...
Read more >
Environment Variables in Next.js - Frontend Digest
This creates an environment variable called GREETING which contains the string Hello world . Note that you need to restart Next.js to apply...
Read more >
Accenture | Let there be change
Using technology and human ingenuity to meet environmental goals for a powerful, sustainable future. explore. Careers Blog. My career path to digital ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found