AzureLogHandler does not seem to work from Celery tasks

Python 3.5.2
Django 2.2.10
celery==4.2.2
opencensus==0.7.7
opencensus-context==0.1.1
opencensus-ext-azure==1.0.2
opencensus-ext-django==0.7.2
opencensus-ext-logging==0.1.0

After a loooong investigation and many experiments with various ways of configuring logging in Celery tasks and django, I am pretty sure that for some reason, the AzureLogHandler does not send any log to Azure from a task running in a Celery worker.

To make sure of that, in my django project, I setup Celery logging by discarding the default Celery logging behaviour (the default is misleading because it overrides the root logger) and instead I use the django logging configuration (from django’s settings.py), and in addition I manually make sure the celery logger has both a local file log handler (making sure I capture all log messages locally for debugging this issue) and a AzureLogHandler. You can see how this is configuring in the code snippet below:

import os
import logging

from celery import Celery
from celery.signals import setup_logging
from opencensus.ext.azure.log_exporter import AzureLogHandler

from django.conf import settings


os.environ.setdefault("DJANGO_SETTINGS_MODULE", "edgefolio.settings")

# Create the Celery app (called "api")
app = Celery('api')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)


def add_azure_log_handler_to_logger(logger, propagate=True):
    """
    Given a logger, add a AzureLogHandler to it
    :param logger:
    :param propagate:
    :return:
    """
    if settings.AZURE_APP_INSIGHTS_INSTR_KEY:
        formatter = logging.Formatter("[Celery/%(processName)s] %(message)s")
        # Azure Handler:
        azure_log_handler = AzureLogHandler(
            connection_string='InstrumentationKey=%s' % settings.AZURE_APP_INSIGHTS_INSTR_KEY)
        azure_log_handler.setFormatter(formatter)
        azure_log_handler.setLevel(logging.INFO)
        logger.addHandler(azure_log_handler)
        logger.setLevel(logging.INFO)
        logger.propagate = propagate
    return logger


@setup_logging.connect
def setup_loggers(*args, **kwargs):
    """
    Using the celery "setup_logging" signal to override and fully define the logging configuration for Celery
    :param args:
    :param kwargs:
    :return:
    """
    # Configure Celery logging from the Django settings' logging configuration
    from logging.config import dictConfig
    from django.conf import settings
    dictConfig(settings.LOGGING)

    # Test the root logger (configured in django settings to log to Azure as well
    logger = logging.getLogger('root')
    logger.warning('TRYING LOGGING FROM [%s]' % logger.name)

    # Configure the Celery top level logger
    logger = logging.getLogger('celery')
    # Add a local file log handler to make sure we capture every message locally
    logger.addHandler(logging.FileHandler("/data/log/worker/importer.log"))
    # In addition, also manually add a AzureLogHandler to it (duplicate with the root's handler)
    logger = add_azure_log_handler_to_logger(logger, propagate=False)
    # Log a test warning message
    logger.warning('TRYING LOGGING FROM [%s]' % logger.name)

    # Log a test warning message from a lower-level celery logger
    logger = logging.getLogger('celery.task')
    logger.warning('TRYING LOGGING FROM [%s]' % logger.name)

    # Log a test warning message from a specific django app task logger
    logger = logging.getLogger('etl.tasks')
    logger.warning('TRYING LOGGING FROM [%s]' % logger.name)

The result of this setup is that the log messages logged from the main celery process are all sent to both the local file and to Azure Monitor without problem, however log messages generated from inside the celery tasks are sent to the local file (proving that the logging configuration above is used in the tasks as well) but NOT to Azure Monitor. It appears only logs from the main Celery process are sent to Azure, but none from the Celery Worker processes.

I wonder if this has to do with the way AzureLogHandler is implemented (using threading and similar wizardry) and the way Celery worker processes run.

[trevorphillipscoding]

AzureLogHandler doesn’t work with celery for me either. Any update on this?

[lzchen]

@SanthoshMedide Can you open up a new issue for this? It seems to be unrelated to this thread.