Migrations problem with custom logger

Apache Airflow version: 2.0.2

Kubernetes version : v1.19.12

Environment:

• Cloud provider or hardware configuration: hardware
• OS (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
• Kernel (e.g. uname -a): 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 GNU/Linux
• Install tools: Helm
• Others:

What happened:

We tried to make custom logger like in documentation, because we want redirect task logs to stdout (by default logs writes to file in worker pod). We made code for custom logger, set param logging_config_class = logconfig.task_logs_to_std.LOGGING_CONFIG, set PYTHONPATH in helm chart. After that, we created configMap with logger source and mounted it inside pod with volume+volumeMount to directory logconfig When we installed this chart, there was a error with run-airflow-migrations.

ERROR! Maximum number of retries (20) reached.

 

Last check result:
$ airflow db check
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/logging_config.py", line 40, in configure_logging
    logging_config = import_string(logging_class_path)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/module_loading.py", line 32, in import_string
    module = import_module(module_path)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'logconfig'

 

During handling of the above exception, another exception occurred:

 

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__init__.py", line 46, in <module>
    settings.initialize()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/settings.py", line 434, in initialize
    LOGGING_CLASS_PATH = configure_logging()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/logging_config.py", line 49, in configure_logging
    raise ImportError(f'Unable to load custom logging from {logging_class_path} due to {err}')
ImportError: Unable to load custom logging from logconfig.task_logs_to_std.LOGGING_CONFIG due to No module named 'logconfig'

In result, pods with scheduler and webserver does not works.

Logger code:

from copy import deepcopy
from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
import sys

LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
LOGGING_CONFIG["handlers"]["task"] = {
    "class": "logging.StreamHandler",
    "formatter": "airflow",
    "stream": sys.stdout,
}

What you expected to happen: I expected that migration will be ok and we can see task logs in stdout.

Anything else we need to know: If there are any way how to redirect task logs in pods to stdout - please let me know.

[potiuk]

Apparently you miss logconfig module in your image. You need to create a custom image and make sure logconfig is instaled there. See https://airflow.apache.org/docs/docker-stack/build.html on how to build your own image with your dependencies.

[WolfWood72]

Thank you for answer!