bug: strict_env_var_mode feature flag not respected?

Meltano Version

2.10

Python Version

3.7

Bug scope

Configuration (settings parsing, validation, etc.)

Operating System

Amaon Linux

Description

I think the change in strict_env_var_mode that came out w 2.10 (https://github.com/meltano/meltano/pull/6971) is causing issues for me with pipelinewise/target-snowflake. Not 100% sure if its a real bug or something’s messed up in my setup that was revealed by that change. But upon upgrade to 2.10, all my runs started failing with "Environment variable 'TARGET_SNOWFLAKE_SCHEMA' referenced but not set. I don’t think I’ve ever set that variable and I dont… think(?) that plugin is actually expecting that env var? I am also noticing that this error gets thrown only for schedules that are defined in the “legacy ETL” syntax, not via “jobs”.

Code

2022-11-16T22:36:29.972001Z [error    ] ("Environment variable 'TARGET_SNOWFLAKE_SCHEMA' referenced but not set", 'Make sure the environment variable is set') 
Traceback (most recent call last):
  File "/home/ec2-user/meltano/venv/lib64/python3.7/site-packages/meltano/core/utils/__init__.py", line 495, in subst
    val = str(env[var])
KeyError: 'TARGET_SNOWFLAKE_SCHEMA'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/meltano/venv/lib64/python3.7/site-packages/meltano/core/logging/output_logger.py", line 201, in redirect_logging
    yield
  File "/home/ec2-user/meltano/venv/lib64/python3.7/site-packages/meltano/cli/elt.py", line 257, in _redirect_output
    yield
  File "/home/ec2-user/meltano/venv/lib64/python3.7/site-packages/meltano/cli/elt.py", line 271, in _run_elt
    elt_context = context_builder.context()
  File "/home/ec2-user/meltano/venv/lib64/python3.7/site-packages/meltano/core/elt_context.py", line 467, in context
    env.update(loader.env)
  File "/home/ec2-user/meltano/venv/lib64/python3.7/site-packages/meltano/core/elt_context.py", line 77, in env
    return {**self.plugin.info_env, **self.config_env()}
  File "/home/ec2-user/meltano/venv/lib64/python3.7/site-packages/meltano/core/elt_context.py", line 68, in config_env
    return self.settings_service.as_env(session=self.session, **kwargs)
  File "/home/ec2-user/meltano/venv/lib64/python3.7/site-packages/meltano/core/settings_service.py", line 287, in as_env
    for _, config in self.config_with_metadata(*args, **kwargs).items():
  File "/home/ec2-user/meltano/venv/lib64/python3.7/site-packages/meltano/core/settings_service.py", line 240, in config_with_metadata
    **kwargs,
  File "/home/ec2-user/meltano/venv/lib64/python3.7/site-packages/meltano/core/settings_service.py", line 362, in get_with_metadata
    value, env=expandable_env, raise_if_missing=env_var_strict_mode
  File "/home/ec2-user/meltano/venv/lib64/python3.7/site-packages/meltano/core/utils/__init__.py", line 511, in expand_env_vars
    return subst(fullmatch)
  File "/home/ec2-user/meltano/venv/lib64/python3.7/site-packages/meltano/core/utils/__init__.py", line 504, in subst
    raise EnvironmentVariableNotSetError(e.args[0])
meltano.core.utils.EnvironmentVariableNotSetError: ("Environment variable 'TARGET_SNOWFLAKE_SCHEMA' referenced but not set", 'Make sure the environment variable is set')

[qbatten]

1. config dump & grep.

(venv) [ec2-user@redact meltano-main]$ meltano config meltano | grep "strict_env_var_mode"
2022-11-17T17:24:06.807913Z [info     ] Environment 'prod' is active
    "strict_env_var_mode": false`

1. Wasn’t able to get to this yet, I figured I’d post this for now anyway

2. It’s not set in the .env file, nor in meltano.yml. Just queried the system DB and the plugin_settings table has 0 rows.

[qbatten]

Actually this problem does seem to be occurring with jobs too.

I worked around it by setting TARGET_SNOWFLAKE_SCHEMA in my dotenv file. The variable isn’t actually used by pipelinewise/target-snowflake so it doesnt affect the schema written to.

Still not sure what the root cause is, but it is definitely still happening with MELTANO_FF_STRICT_ENV_VAR_MODE=false in dotenv. And I also confirmed that the way I’m setting it is indeed changing the config; When I set it to true, the output of meltano config meltano | grep strict does show it changing to true.

Here's a debug error printout (heavily redacted) that happens to also show that strict_env_var_mode is indeed set to false. Maybe this'll help w diagnosis?

 [debug    ] Creating engine '<meltano.core.project.Project object at 0x12341234>@postgresql://xxxxxxxxxxx'
 [info     ] The default environment 'dev' will be ignored for `meltano schedule`. To configure a specific environment, please use the option `--environment=<environment name>`.
 [debug    ] Creating engine '<meltano.core.project.Project object at 12341234>@postgresql://xxxxxxxxxxxxx'
 [info     ] Environment 'dev' is active
 [debug    ] expanding job to tasks         job_name=job-xxxxxxxx-0 tasks=['xxxxxxxx-0', 'target-snowflake-xxxxxxxx']
 [debug    ] Encoding detection: ascii is most likely the one.
 [debug    ] Found plugin parent            parent=tap-postgres plugin=tap-postgres-xxxxxx source=discovery
 [debug    ] Found plugin parent            parent=tap-postgres-xxxxxxxx plugin=xxxxxxxx-0 source=inherited
 [debug    ] found plugin in cli invocation plugin_name=xxxxxxxx-0
 [debug    ] Found plugin parent            parent=target-snowflake plugin=target-snowflake source=discovery
 [debug    ] Found plugin parent            parent=target-snowflake plugin=target-snowflake-xxxxxxxx source=inherited
 [debug    ] found plugin in cli invocation plugin_name=target-snowflake-xxxxxxxx
 [debug    ] head of set is extractor as expected block=<meltano.core.plugin.project_plugin.ProjectPlugin object at 12341234>
 [debug    ] found block                    block_type=loaders index=1
 [debug    ] blocks                         idx=1 offset=0
 [debug    ] Environment variable 'TARGET_SNOWFLAKE_SCHEMA' referenced but not set. Make sure the environment variable is set. 
╭─────────────────────────────── [1;31mTraceback [1;2;31m(most recent call last) ────────────────────────────────╮
│ xxxxxxxxxxxxxxx/venv/lib64/python3.7/site-packages/meltano/core/utils/[1;33m__init__.py:49 │
│ 5 in subst                                                                                       │
│                                                                                                  │
│   492 │   │   try:                                                                               │
│   493 │   │   │   # the variable can be in either group                                          │
│   494 │   │   │   var = next(var for var in match.groups() if var)                               │
│ ❱ 495 │   │   │   val = str(env[var])                                                            │
│   496 │   │   │                                                                                  │
│   497 │   │   │   if not val:                                                                    │
│   498 │   │   │   │   logger.debug(f"Variable '${var}' is empty.")                               │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │              env = {                                                                         │ │
│ │                    │   'MELTANO_FF_STRICT_ENV_VAR_MODE': 'false',                            │ │
│ │                    │   'MELTANO_DATABASE_URI':'postgresql:// xxxxxxxxxxxxxxxxxxxxxxxxxxxx    │ │
│ │                    │   ... +123                                                              │ │
│ │                    }                                                                         │ │
│ │            match = <[1;95mre.Match object; span=[1;39m(0, 24[1;39m), match='$TARGET_SNOWFLAKE_SCHEMA'>         │ │
│ │ raise_if_missing = 'false'                                                                   │ │
│ │              var = 'TARGET_SNOWFLAKE_SCHEMA'                                                 │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
[1;91mKeyError: 'TARGET_SNOWFLAKE_SCHEMA'