First exception not flushed on Celery+Python until second occurrence

Here is an issue that’s a little hard to understand. We run Sentry on Django+Celery.

Django==2.2.12
celery==4.4.2
sentry-sdk==0.14.3

We run many packages on that project, so I suspect it’s a conflict with another one but I do not know where to start.

Details

1. Exceptions are reported to Sentry as expected on the Django wsgi
2. Exceptions from code running on Celery will not be reported immediately
3. Triggering the same exception (same fingerprint, different message) a second time will “flush” both and they will both appear on Sentry
4. Triggering 2 different exceptions on a single Celery task will not report either of them to Sentry
5. Calling Hub.current.client.flush() doesn’t change any of it

Celery task

@app.task
def sentry_logging_task(what):
    try:
        raise Exception("Sentry say what? {}".format(what))
    except Exception as e:
        logger.exception(str(e))

Sentry init

sentry_sdk.init(
    dsn=settings.DNS,
    integrations=[DjangoIntegration(), CeleryIntegration()],
    environment=settings.ENV
 )
 ignore_logger('django.security.DisallowedHost')

[RobbieClarken]

I’ve confirmed that this bug is still present in sentry-sdk 1.5.1 and master although the behaviour has changed due to commit a6cc9718fe398acee134e6ee9297e0fddea9b359.

There is still the issue that the first logged error doesn’t get processed immediately and could stay pending on the queue indefinitely. However:

• Prior to a6cc9718fe398acee134e6ee9297e0fddea9b359 when the Celery worker was terminated the pending event would be dropped and never reported.
• From a6cc9718fe398acee134e6ee9297e0fddea9b359 onwards when the Celery worker is terminated the pending event does get reported to Sentry.

I still consider this an issue because the Celery worker may not be terminated for quite some time and so the error may not be reported until it is too late.

The code in https://github.com/getsentry/sentry-python/issues/687#issuecomment-837738001 can still be used to replicate the issue. By default Celery autoscales down inactive workers after 30 seconds so the error will be reported after 30 seconds. This can be changed with the AUTOSCALE_KEEPALIVE environment variable. For example, setting AUTOSCALE_KEEPALIVE=600 will demonstrate the error doesn’t get reported for 10 minutes.

[github-actions[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone … forever!

---

“A weed is but an unloved flower.” ― Ella Wheeler Wilcox 🥀