Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

celery SQLAlchemy errors in celery tasks that use sqlalchemy pools

See original GitHub issue

While running multiple alert.run_query celery tasks, some tasks fail with a sqlalchemy.exc.NoSuchColumnError error here https://github.com/apache/incubator-superset/blob/ece91928a9339190163c0bc72b96e51217a90d1e/superset/tasks/schedules.py#L544 on the first few iterations of the celery beat schedule after its initialization. These errors eventually stop occurring if the celery worker and the beat schedule are left to run for a few iterations without stopping. Tests have shown that the errors stop occurring on average after around 2 iterations of the beat schedule.

Expected results

Alerting celery tasks run without error

Actual results

Traceback (most recent call last):
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/thirdparty/superset_python3/pip/celery/celery-cpython-38/lib/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/thirdparty/superset_python3/superset-cpython-38/lib/superset/app.py", line 115, in __call__
    return task_base.__call__(self, *args, **kwargs)
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/thirdparty/superset_python3/pip/celery/celery-cpython-38/lib/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/thirdparty/superset_python3/superset-cpython-38/lib/superset/tasks/schedules.py", line 562, in schedule_alert_query
    if run_alert_query(
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/thirdparty/superset_python3/superset-cpython-38/lib/superset/tasks/schedules.py", line 672, in run_alert_query
    deliver_alert(alert_id)
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/thirdparty/superset_python3/superset-cpython-38/lib/superset/tasks/schedules.py", line 579, in deliver_alert
    alert = scoped_session.query(Alert).get(alert_id)
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/query.py", line 1021, in get
    return self._get_impl(ident, loading.load_on_pk_identity)
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/query.py", line 1138, in _get_impl
    return db_load_fn(self, primary_key_identity)
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/loading.py", line 287, in load_on_pk_identity
    return q.one()
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/query.py", line 3458, in one
    ret = self.one_or_none()
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/query.py", line 3427, in one_or_none
    ret = list(self)
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/loading.py", line 101, in instances
    cursor.close()
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/loading.py", line 81, in instances
    rows = [proc(row) for row in fetch]
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/loading.py", line 81, in <listcomp>
    rows = [proc(row) for row in fetch]
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/loading.py", line 525, in _instance
    tuple([row[column] for column in pk_cols]),
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/orm/loading.py", line 525, in <listcomp>
    tuple([row[column] for column in pk_cols]),
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/engine/result.py", line 681, in _key_fallback
    util.raise_(
  File "/srv/yaps/mounts/superset/superset_build/bbe899d629f9d61e5d2a8ffdccbcc57724a557ea-e230e418e5960bc45d735cb21d806661/celery.runfiles/__main__/pip/SQLAlchemy/SQLAlchemy-cpython-38/lib/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
sqlalchemy.exc.NoSuchColumnError: "Could not locate column in row for column 'alerts.id'"

How to reproduce the bug

install geckodriver
set up config.py to enable alerting

ENABLE_ALERTS = True
FEATURE_FLAGS = {"THUMBNAILS": True}

THUMBNAIL_SELENIUM_USER = "Admin"
celery_alerting = {'alerts.schedule_check': {
        'task': 'alerts.schedule_check',
        'schedule': crontab(minute='*/1'),
    }}
CeleryConfig.CELERYBEAT_SCHEDULE.update(celery_alerting)
CELERY_CONFIG = CeleryConfig

Go to Manage -> Alerts
Create a few alerts (~6) that will run at the same time
Start a celery worker and celery beat
See error in celery worker

Environment

superset version: 0.37
python version: 3.7.7

Checklist

Make sure these boxes are checked before submitting your issue - thank you!

I have checked the superset logs for python stacktraces and included it here as text if there are any.
I have reproduced the issue with at least the latest released version of superset.
I have checked the issue tracker for the same issue and I haven’t found one similar.

Issue Analytics

State:
Created 3 years ago
Comments:10 (6 by maintainers)

Top GitHub Comments

1reaction

robdiciucciocommented, Feb 25, 2021

I’m working on an app-level fix for this based on https://www.yangster.ca/post/not-the-same-pre-fork-worker-model/. This should also resolve https://github.com/apache/superset/issues/12766

0reactions

robdiciucciocommented, Feb 25, 2021

https://github.com/apache/superset/pull/13350 contains a potential fix for this issue. Can you please test and confirm?

Top Results From Across the Web

celery SQLAlchemy errors in celery tasks that use ... - GitHub

It seems that celery is not happy to deal with the sqlalchemy pool & that's related to the process forking. The recommendation I...

SqlAlchemy + Celery with Scoped Session error

I fixed this by using Celery events to invalidate all existing ... More info: https://docs.sqlalchemy.org/en/latest/core/pooling.html#using- ...

Using SQLAlchemy with Celery Tasks - prschmid

Let's update our tasks module with a new task that gets something from the database.

Connection Pooling - SQLAlchemy 1.4 Documentation

A connection pool is a standard technique used to maintain long running connections in memory for efficient re-use, as well as to provide ......

Configuration and defaults — Celery 5.2.7 documentation

Default: {} (empty mapping). When SQLAlchemy is configured as the result backend, Celery automatically creates two tables to store result meta-data for tasks....