bug: Airflow causes meltano to print log every few seconds

Meltano Version

2.4.0

Python Version

3.9

Bug scope

CLI (options, error messages, logging, etc.)

Operating System

macOS - Mojave

Description

When we execute meltano invoke airflow scheduler, meltano emits a line of log displaying selected environment every few seconds. I believe the root cause is:

1. Airflow tries to update DAG every few seconds, calling orchestrate/dags/meltano.py
2. The script invokes meltano CLI: meltano schedule list --format=json to retrieve the schedules
3. Meltano CLI prints the line: Environment ... is active

Here goes the orchestrators part of my meltano.yml:

  orchestrators:
  - name: airflow
    variant: apache
    pip_url: apache-airflow==2.1.2 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-${MELTANO__PYTHON_VERSION}.txt
  files:
  - name: files-airflow
    variant: meltano
    pip_url: git+https://github.com/meltano/files-airflow.git

It’s an issue derived by side effects combined together. IMHO, the best approach (and a legit enhancement) is probably to provide a way to suppress the log message (maybe a --slient switch) with meltano schedule list command, and then we can update the script in files-airflow to turn it off when retrieving the schedules.

Although it’s also possible to suppress the log in console by passing stderr=subprocess.DEVNULL in files-airflow script, it won’t really solve the issue, for it still generates the log and flows into logging handlers other than console.

Currently, I can only work this around by raising the log level of the CLI logger. In logging.yaml:

#...
loggers:
  meltano.cli:
    level: ERROR
    handlers: [console, file]
#...

Code

No response

[pandemicsyn]

Unfortunately I had my own log config file which I believe takes precedence over --log-level so it didn’t succeed to suppressed the message.

@simonpai It’s not ideal, but you could run two logging configs. One for regular meltano invocations as you are today and one for use by the airflow dag generator, and instruct the dag generator to use the alternate one via the --log-config flag.

[aaronsteers]

Re:

I had my own log config file which I believe takes precedence over --log-level so it didn’t succeed to suppressed the message.

Should we change precedence order? Seems like CLI flag having precedence over the config file would make sense.