Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

bug: env inheritance global env -> plugin env

See original GitHub issue

Prereq:

#6876

Meltano Version

2.9.0

Python Version

3.8

Bug scope

Configuration (settings parsing, validation, etc.)

Operating System

Ubuntu

Description

Copy meltano.yml
meltano install
export MELTY=melty
`meltano invoke --print-var GLOBALMELTY–print-var DEVMELTY target-jsonl

visch@visch-ubuntu:~/git/meltano-projects/env_multiple_values_inheritance$ meltano invoke --print-var DEVMELTY --print-var GLOBALMELTY target-jsonl
2022-11-16T02:46:06.088209Z [info     ] Environment 'dev' is active
DEVMELTY=MELTY
GLOBALMELTY=$MELTY

Code

version: 1
default_environment: dev
project_id: bea6b7f5-2f98-416a-9a6b-b602b56efff6
environments:
- name: dev
  env:
    DEVMELTY: $MELTY
- name: staging
- name: prod
plugins:
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
env:
  GLOBALMELTY: $MELTY

Issue Analytics

State:
Created 10 months ago
Reactions:2
Comments:6 (1 by maintainers)

Top GitHub Comments

3reactions

cjohnhansoncommented, Nov 16, 2022

@tayloramurphy – yes we have these tests for inheritance.

The logic itself is consolidated in plugin_invoker.py and is well-commented there.

I was able to replicate this locally. I think the fix itself is fairly straightforward – it looks like we’re expanding the top-level env vars in the plugin invoker, but then the plugin itself is still getting the unexpanded top-level values in its env and those take precedence over top-level env vars when we pass the full environment into the plugin at invocation. So scope here would be to update that precedence order and/or potentially expand env vars with plugin settings directly and then figure out why this behavior isn’t cropping up in unit tests and update the test suite to catch it (probably also add an integration test for this as well)

1reaction

tayloramurphycommented, Nov 16, 2022

@cjohnhanson thanks for the details! Can you give this a prelim weight on the Product Roadmap project and I’ll work with AJ to get it prioritized?