question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Async queries failing in celery workers

See original GitHub issue

When running a query in SQL Lab against a database with async queries enabled, the following error is raised by the celery worker:

Task sql_lab.get_sql_results[051acb0d-30a5-4dbe-b30e-5b16bc9d8545] raised unexpected: RuntimeError('Working outside of application context.\n\nThis typically means that you attempted to use functionality that needed\nto interface with the current application object in some way. To solve\nthis, set up an application context with app.app_context().  See the\ndocumentation for more information.',)
Traceback (most recent call last):
  File "/Users/rob/work/incubator-superset/venv/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/Users/rob/work/incubator-superset/venv/lib/python3.6/site-packages/celery/app/trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "/Users/rob/work/incubator-superset/superset/sql_lab.py", line 154, in get_sql_results
    with session_scope(not ctask.request.called_directly) as session:
  File "/Users/rob/work/incubator-superset/venv/lib/python3.6/site-packages/contextlib2.py", line 79, in __enter__
    return next(self.gen)
  File "/Users/rob/work/incubator-superset/superset/sql_lab.py", line 117, in session_scope
    app.config["SQLALCHEMY_DATABASE_URI"], poolclass=NullPool
  File "/Users/rob/work/incubator-superset/venv/lib/python3.6/site-packages/werkzeug/local.py", line 348, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/Users/rob/work/incubator-superset/venv/lib/python3.6/site-packages/werkzeug/local.py", line 307, in _get_current_object
    return self.__local()
  File "/Users/rob/work/incubator-superset/venv/lib/python3.6/site-packages/flask/globals.py", line 52, in _find_app
    raise RuntimeError(_app_ctx_err_msg)
RuntimeError: Working outside of application context.

This typically means that you attempted to use functionality that needed
to interface with the current application object in some way. To solve
this, set up an application context with app.app_context().  See the
documentation for more information.

Expected results

Celery worker should run the query and store the results in the RESULTS_BACKEND.

Actual results

Exception raised (see above).

Screenshots

If applicable, add screenshots to help explain your problem.

How to reproduce the bug

  1. Go to SQL Lab
  2. Run a query against a database with Asynchronous Query Operation enabled
  3. Note that the UI hangs and a result is never returned
  4. Celery worker raises the above error

Environment

(please complete the following information):

  • superset version: master (acf0753504095a5c0e320b049370aaf6f04e5a4f)
  • python version: 3.6.8
  • node.js version: v12.4.0
  • npm version: 6.9.0

Checklist

Make sure these boxes are checked before submitting your issue - thank you!

  • I have checked the superset logs for python stacktraces and included it here as text if there are any.
  • I have reproduced the issue with at least the latest released version of superset.
  • I have checked the issue tracker for the same issue and I haven’t found one similar.

Additional context

Per @craig-rueda, may be related to application context changes in https://github.com/apache/incubator-superset/pull/8418

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

2reactions
tcopplecommented, Nov 26, 2019

I received the same error when starting up a new environment in docker using docker-compose up. Fresh repo checkout and fresh container setup.

0reactions
tcopplecommented, Nov 27, 2019

Yea, I pulled a previous tag and things were fine. Thanks for the quick response.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Async queries failing in celery workers · Issue #8651
When running a query in SQL Lab against a database with async queries enabled, the following error is raised by the celery worker:...
Read more >
Capture failed tasks from Celery workers during ...
I am currently using Task.delay() in celery (with RabbitMQ) to perform asynchronous tasks. The ...
Read more >
Automatically Retrying Failed Celery Tasks
In this article, we'll look at how to automatically retry failed Celery tasks. Django + Celery Series: Asynchronous Tasks with Django and ...
Read more >
Working with Asynchronous Celery Tasks – lessons learned
Let's move on. Think about the case when render_html function is very slow (e.g., badly implemented). If so, it's possible that race condition ......
Read more >
Retrying Asynchronous Tasks With Celery
Say for instance pagination of a resource is required and a request made in a task fails before completion after a number of...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found