Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

2.0.0b2: SQLAlchemy execute cannot be used while an asynchronous query is underway

See original GitHub issue

Apache Airflow version: 2.0.0b2

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

What happened: Error in webserver logs, relating to Exception on /last_dagruns [POST]

airflow_webserver_1  | [2020-11-17 17:40:31,269] {app.py:1891} ERROR - Exception on /last_dagruns [POST]
airflow_webserver_1  | Traceback (most recent call last):
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
airflow_webserver_1  |     self.dialect.do_execute(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
airflow_webserver_1  |     cursor.execute(statement, parameters)
airflow_webserver_1  | psycopg2.ProgrammingError: execute cannot be used while an asynchronous query is underway
airflow_webserver_1  | 
airflow_webserver_1  | The above exception was the direct cause of the following exception:
airflow_webserver_1  | 
airflow_webserver_1  | Traceback (most recent call last):
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
airflow_webserver_1  |     response = self.full_dispatch_request()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
airflow_webserver_1  |     rv = self.handle_user_exception(e)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
airflow_webserver_1  |     reraise(exc_type, exc_value, tb)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
airflow_webserver_1  |     raise value
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
airflow_webserver_1  |     rv = self.dispatch_request()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
airflow_webserver_1  |     return self.view_functions[rule.endpoint](**req.view_args)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/auth.py", line 33, in decorated
airflow_webserver_1  |     if appbuilder.sm.check_authorization(permissions, request.args.get('dag_id', None)):
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 679, in check_authorization
airflow_webserver_1  |     can_access_all_dags = self.has_access(*perm)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 347, in has_access
airflow_webserver_1  |     has_access |= self.can_read_dag(resource, user)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 293, in can_read_dag
airflow_webserver_1  |     return self._has_view_access(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/security/manager.py", line 1117, in _has_view_access
airflow_webserver_1  |     return self.exist_permission_on_roles(view_name, permission_name, db_role_ids)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/security/sqla/manager.py", line 337, in exist_permission_on_roles
airflow_webserver_1  |     return self.appbuilder.get_session.query(q).scalar()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3491, in scalar
airflow_webserver_1  |     ret = self.one()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3458, in one
airflow_webserver_1  |     ret = self.one_or_none()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3427, in one_or_none
airflow_webserver_1  |     ret = list(self)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3503, in __iter__
airflow_webserver_1  |     return self._execute_and_instances(context)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3528, in _execute_and_instances
airflow_webserver_1  |     result = conn.execute(querycontext.statement, self._params)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1014, in execute
airflow_webserver_1  |     return meth(self, multiparams, params)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
airflow_webserver_1  |     return connection._execute_clauseelement(self, multiparams, params)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1127, in _execute_clauseelement
airflow_webserver_1  |     ret = self._execute_context(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
airflow_webserver_1  |     self._handle_dbapi_exception(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
airflow_webserver_1  |     util.raise_(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
airflow_webserver_1  |     raise exception
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
airflow_webserver_1  |     self.dialect.do_execute(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
airflow_webserver_1  |     cursor.execute(statement, parameters)
airflow_webserver_1  | sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) execute cannot be used while an asynchronous query is underway
airflow_webserver_1  | [SQL: SELECT EXISTS (SELECT 1 
airflow_webserver_1  | FROM ab_permission_view JOIN ab_permission_view_role ON ab_permission_view.id = ab_permission_view_role.permission_view_id JOIN ab_role ON ab_role.id = ab_permission_view_role.role_id JOIN ab_permission ON ab_permission.id = ab_permission_view.permission_id JOIN ab_view_menu ON ab_view_menu.id = ab_permission_view.view_menu_id 
airflow_webserver_1  | WHERE ab_view_menu.name = %(name_1)s AND ab_permission.name = %(name_2)s AND ab_role.id IN (%(id_1)s)) AS anon_1]
airflow_webserver_1  | [parameters: {'name_1': 'DAGs', 'name_2': 'can_read', 'id_1': 1}]
airflow_webserver_1  | (Background on this error at: http://sqlalche.me/e/13/f405)
airflow_webserver_1  | [2020-11-17 17:40:39 +0000] [43] [INFO] Handling signal: ttin
airflow_webserver_1  | [2020-11-17 17:40:39 +0000] [63] [INFO] Booting worker with pid: 63
airflow_webserver_1  | /home/airflow/.local/lib/python3.8/site-packages/airflow/www/gunicorn_config.py:
airflow_webserver_1  | 31 DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
airflow_webserver_1  | [2020-11-17 17:40:46 +0000] [43] [INFO] Handling signal: ttou
airflow_webserver_1  | [2020-11-17 17:42:31 +0000] [43] [CRITICAL] WORKER TIMEOUT (pid:46)
airflow_webserver_1  | [2020-11-17 17:42:31 +0000] [43] [CRITICAL] WORKER TIMEOUT (pid:57)
airflow_webserver_1  | [2020-11-17 17:42:32 +0000] [69] [INFO] Booting worker with pid: 69
airflow_webserver_1  | 172.24.0.1 - - [17/Nov/2020:17:42:32 +0000] "GET /static/pin_32.png HTTP/1.1" 200 0 "http://localhost:8080/home" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
airflow_webserver_1  | [2020-11-17 17:42:32,311] {app.py:1891} ERROR - Exception on /dag_stats [POST]
airflow_webserver_1  | Traceback (most recent call last):
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
airflow_webserver_1  |     response = self.full_dispatch_request()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
airflow_webserver_1  |     rv = self.handle_user_exception(e)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
airflow_webserver_1  |     reraise(exc_type, exc_value, tb)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
airflow_webserver_1  |     raise value
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
airflow_webserver_1  |     rv = self.dispatch_request()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
airflow_webserver_1  |     return self.view_functions[rule.endpoint](**req.view_args)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/auth.py", line 33, in decorated
airflow_webserver_1  |     if appbuilder.sm.check_authorization(permissions, request.args.get('dag_id', None)):
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 679, in check_authorization
airflow_webserver_1  |     can_access_all_dags = self.has_access(*perm)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 339, in has_access
airflow_webserver_1  |     if user.is_anonymous:
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/local.py", line 347, in __getattr__
airflow_webserver_1  |     return getattr(self._get_current_object(), name)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/local.py", line 306, in _get_current_object
airflow_webserver_1  |     return self.__local()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/utils.py", line 26, in <lambda>
airflow_webserver_1  |     current_user = LocalProxy(lambda: _get_user())
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/utils.py", line 335, in _get_user
airflow_webserver_1  |     current_app.login_manager._load_user()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/login_manager.py", line 359, in _load_user
airflow_webserver_1  |     return self.reload_user()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/login_manager.py", line 321, in reload_user
airflow_webserver_1  |     user = self.user_callback(user_id)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/security/manager.py", line 1674, in load_user
airflow_webserver_1  |     return self.get_user_by_id(int(pk))
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/security/sqla/manager.py", line 233, in get_user_by_id
airflow_webserver_1  |     return self.get_session.query(self.user_model).get(pk)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1021, in get
airflow_webserver_1  |     return self._get_impl(ident, loading.load_on_pk_identity)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1138, in _get_impl
airflow_webserver_1  |     return db_load_fn(self, primary_key_identity)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 287, in load_on_pk_identity
airflow_webserver_1  |     return q.one()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3458, in one
airflow_webserver_1  |     ret = self.one_or_none()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3427, in one_or_none
airflow_webserver_1  |     ret = list(self)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3503, in __iter__
airflow_webserver_1  |     return self._execute_and_instances(context)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3524, in _execute_and_instances
airflow_webserver_1  |     conn = self._get_bind_args(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3539, in _get_bind_args
airflow_webserver_1  |     return fn(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3518, in _connection_from_session
airflow_webserver_1  |     conn = self.session.connection(**kw)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1138, in connection
airflow_webserver_1  |     return self._connection_for_bind(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1146, in _connection_for_bind
airflow_webserver_1  |     return self.transaction._connection_for_bind(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 440, in _connection_for_bind
airflow_webserver_1  |     if self.session.twophase and self._parent is None:
airflow_webserver_1  | AttributeError: 'NoneType' object has no attribute 'twophase'
airflow_webserver_1  | 172.24.0.1 - - [17/Nov/2020:17:42:32 +0000] "POST /dag_stats HTTP/1.1" 500 5322 "http://localhost:8080/home" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
airflow_webserver_1  | [2020-11-17 17:42:32,354] {app.py:1891} ERROR - Exception on /last_dagruns [POST]
airflow_webserver_1  | Traceback (most recent call last):
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
airflow_webserver_1  |     response = self.full_dispatch_request()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
airflow_webserver_1  |     rv = self.handle_user_exception(e)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
airflow_webserver_1  |     reraise(exc_type, exc_value, tb)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
airflow_webserver_1  |     raise value
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
airflow_webserver_1  |     rv = self.dispatch_request()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
airflow_webserver_1  |     return self.view_functions[rule.endpoint](**req.view_args)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/auth.py", line 33, in decorated
airflow_webserver_1  |     if appbuilder.sm.check_authorization(permissions, request.args.get('dag_id', None)):
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 679, in check_authorization
airflow_webserver_1  |     can_access_all_dags = self.has_access(*perm)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 339, in has_access
airflow_webserver_1  |     if user.is_anonymous:
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/local.py", line 347, in __getattr__
airflow_webserver_1  |     return getattr(self._get_current_object(), name)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/local.py", line 306, in _get_current_object
airflow_webserver_1  |     return self.__local()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/utils.py", line 26, in <lambda>
airflow_webserver_1  |     current_user = LocalProxy(lambda: _get_user())
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/utils.py", line 335, in _get_user
airflow_webserver_1  |     current_app.login_manager._load_user()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/login_manager.py", line 359, in _load_user
airflow_webserver_1  |     return self.reload_user()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/login_manager.py", line 321, in reload_user
airflow_webserver_1  |     user = self.user_callback(user_id)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/security/manager.py", line 1674, in load_user
airflow_webserver_1  |     return self.get_user_by_id(int(pk))
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/security/sqla/manager.py", line 233, in get_user_by_id
airflow_webserver_1  |     return self.get_session.query(self.user_model).get(pk)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1021, in get
airflow_webserver_1  |     return self._get_impl(ident, loading.load_on_pk_identity)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1138, in _get_impl
airflow_webserver_1  |     return db_load_fn(self, primary_key_identity)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 287, in load_on_pk_identity
airflow_webserver_1  |     return q.one()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3458, in one
airflow_webserver_1  |     ret = self.one_or_none()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3427, in one_or_none
airflow_webserver_1  |     ret = list(self)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3503, in __iter__
airflow_webserver_1  |     return self._execute_and_instances(context)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3524, in _execute_and_instances
airflow_webserver_1  |     conn = self._get_bind_args(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3539, in _get_bind_args
airflow_webserver_1  |     return fn(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3518, in _connection_from_session
airflow_webserver_1  |     conn = self.session.connection(**kw)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1138, in connection
airflow_webserver_1  |     return self._connection_for_bind(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1146, in _connection_for_bind
airflow_webserver_1  |     return self.transaction._connection_for_bind(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 440, in _connection_for_bind
airflow_webserver_1  |     if self.session.twophase and self._parent is None:
airflow_webserver_1  | AttributeError: 'NoneType' object has no attribute 'twophase'
airflow_webserver_1  | 172.24.0.1 - - [17/Nov/2020:17:42:32 +0000] "POST /last_dagruns HTTP/1.1" 500 5322 "http://localhost:8080/home" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
airflow_webserver_1  | [2020-11-17 17:42:32,376] {app.py:1891} ERROR - Exception on /blocked [POST]
airflow_webserver_1  | Traceback (most recent call last):
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
airflow_webserver_1  |     response = self.full_dispatch_request()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
airflow_webserver_1  |     rv = self.handle_user_exception(e)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
airflow_webserver_1  |     reraise(exc_type, exc_value, tb)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
airflow_webserver_1  |     raise value
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
airflow_webserver_1  |     rv = self.dispatch_request()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
airflow_webserver_1  |     return self.view_functions[rule.endpoint](**req.view_args)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/auth.py", line 33, in decorated
airflow_webserver_1  |     if appbuilder.sm.check_authorization(permissions, request.args.get('dag_id', None)):
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 679, in check_authorization
airflow_webserver_1  |     can_access_all_dags = self.has_access(*perm)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 339, in has_access
airflow_webserver_1  |     if user.is_anonymous:
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/local.py", line 347, in __getattr__
airflow_webserver_1  |     return getattr(self._get_current_object(), name)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/local.py", line 306, in _get_current_object
airflow_webserver_1  |     return self.__local()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/utils.py", line 26, in <lambda>
airflow_webserver_1  |     current_user = LocalProxy(lambda: _get_user())
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/utils.py", line 335, in _get_user
airflow_webserver_1  |     current_app.login_manager._load_user()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/login_manager.py", line 359, in _load_user
airflow_webserver_1  |     return self.reload_user()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/login_manager.py", line 321, in reload_user
airflow_webserver_1  |     user = self.user_callback(user_id)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/security/manager.py", line 1674, in load_user
airflow_webserver_1  |     return self.get_user_by_id(int(pk))
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/security/sqla/manager.py", line 233, in get_user_by_id
airflow_webserver_1  |     return self.get_session.query(self.user_model).get(pk)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1021, in get
airflow_webserver_1  |     return self._get_impl(ident, loading.load_on_pk_identity)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1138, in _get_impl
airflow_webserver_1  |     return db_load_fn(self, primary_key_identity)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 287, in load_on_pk_identity
airflow_webserver_1  |     return q.one()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3458, in one
airflow_webserver_1  |     ret = self.one_or_none()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3427, in one_or_none
airflow_webserver_1  |     ret = list(self)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3503, in __iter__
airflow_webserver_1  |     return self._execute_and_instances(context)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3524, in _execute_and_instances
airflow_webserver_1  |     conn = self._get_bind_args(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3539, in _get_bind_args
airflow_webserver_1  |     return fn(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3518, in _connection_from_session
airflow_webserver_1  |     conn = self.session.connection(**kw)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1138, in connection
airflow_webserver_1  |     return self._connection_for_bind(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1146, in _connection_for_bind
airflow_webserver_1  |     return self.transaction._connection_for_bind(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 440, in _connection_for_bind
airflow_webserver_1  |     if self.session.twophase and self._parent is None:
airflow_webserver_1  | AttributeError: 'NoneType' object has no attribute 'twophase'
airflow_webserver_1  | [2020-11-17 17:42:32,383] {app.py:1891} ERROR - Exception on /task_stats [POST]
airflow_webserver_1  | Traceback (most recent call last):
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
airflow_webserver_1  |     response = self.full_dispatch_request()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
airflow_webserver_1  |     rv = self.handle_user_exception(e)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
airflow_webserver_1  |     reraise(exc_type, exc_value, tb)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
airflow_webserver_1  |     raise value
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
airflow_webserver_1  |     rv = self.dispatch_request()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
airflow_webserver_1  |     return self.view_functions[rule.endpoint](**req.view_args)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/auth.py", line 33, in decorated
airflow_webserver_1  |     if appbuilder.sm.check_authorization(permissions, request.args.get('dag_id', None)):
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 679, in check_authorization
airflow_webserver_1  |     can_access_all_dags = self.has_access(*perm)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/security.py", line 339, in has_access
airflow_webserver_1  |     if user.is_anonymous:
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/local.py", line 347, in __getattr__
airflow_webserver_1  |     return getattr(self._get_current_object(), name)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/local.py", line 306, in _get_current_object
airflow_webserver_1  |     return self.__local()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/utils.py", line 26, in <lambda>
airflow_webserver_1  |     current_user = LocalProxy(lambda: _get_user())
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/utils.py", line 335, in _get_user
airflow_webserver_1  |     current_app.login_manager._load_user()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/login_manager.py", line 359, in _load_user
airflow_webserver_1  |     return self.reload_user()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_login/login_manager.py", line 321, in reload_user
airflow_webserver_1  |     user = self.user_callback(user_id)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/security/manager.py", line 1674, in load_user
airflow_webserver_1  |     return self.get_user_by_id(int(pk))
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/flask_appbuilder/security/sqla/manager.py", line 233, in get_user_by_id
airflow_webserver_1  |     return self.get_session.query(self.user_model).get(pk)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1021, in get
airflow_webserver_1  |     return self._get_impl(ident, loading.load_on_pk_identity)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 1138, in _get_impl
airflow_webserver_1  |     return db_load_fn(self, primary_key_identity)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 287, in load_on_pk_identity
airflow_webserver_1  |     return q.one()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3458, in one
airflow_webserver_1  |     ret = self.one_or_none()
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3427, in one_or_none
airflow_webserver_1  |     ret = list(self)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3503, in __iter__
airflow_webserver_1  |     return self._execute_and_instances(context)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3524, in _execute_and_instances
airflow_webserver_1  |     conn = self._get_bind_args(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3539, in _get_bind_args
airflow_webserver_1  |     return fn(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3518, in _connection_from_session
airflow_webserver_1  |     conn = self.session.connection(**kw)
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1138, in connection
airflow_webserver_1  |     return self._connection_for_bind(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1146, in _connection_for_bind
airflow_webserver_1  |     return self.transaction._connection_for_bind(
airflow_webserver_1  |   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 440, in _connection_for_bind
airflow_webserver_1  |     if self.session.twophase and self._parent is None:
airflow_webserver_1  | AttributeError: 'NoneType' object has no attribute 'twophase'

What you expected to happen: No errors in the webserver logs when /last_dagruns [POST] request is made

How to reproduce it: I used docker compose to test and deploy locally

version: '3.8'
services:
  airflow_postgres:
    image: postgres:latest
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=airflow
      - POSTGRES_DB=airflow
    volumes:
      - "airflow_dbdata:/var/lib/postgresql/data"
    ports:
      - "5432:5432"

  airflow_initdb:
    build:
      context: ..
      dockerfile: docker/Dockerfile
    depends_on:
      - airflow_postgres
    volumes:
      - ../airflow.cfg:/usr/local/airflow/airflow.cfg
      - ../variables:/usr/local/airflow/variables
    command:
      - /bin/bash
      - -c
      - |
        airflow initdb
        if [[ -e /usr/local/airflow/variables/dev/all.json ]]; then
            airflow variables -i /usr/local/airflow/variables/dev/all.json
        fi
        # Enable this if you choose to have RBAC UI activated in the webserver
        # airflow create_user -r Admin -u airflow -e airflow@airflow.com -f Air -l Flow -p airflow

  airflow_webserver:
    build:
      context: ..
      dockerfile: docker/Dockerfile
    restart: always
    depends_on:
      - airflow_initdb
    volumes:
      - ../airflow.cfg:/usr/local/airflow/airflow.cfg
      - ../dags:/usr/local/airflow/dags
      - ../plugins:/usr/local/airflow/plugins
      - ./spark-conf:/spark-conf
    ports:
      - "8080:8080"
      - "4040:4040"
    command: webserver
    healthcheck:
      test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 3

volumes:
  airflow_dbdata:

Anything else we need to know:

I could consistently reproduce this issue. Could it be because of the local metadata store using postgres container? Is it resource constrained in any way ?

Used breeze to build image

./breeze build-image --production-image --python=3.8 --extras=async,password,celery,postgres  --additional-extras=docker,amazon --additional-python-deps="redis statsd" --disable-mysql-client-installation --install-airflow-version="2.0.0b2" --image-tag test-image

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
vikramarsidcommented, Dec 22, 2020

@potiuk I cannot reproduce the above issue with Airflow 2.0.0, 2020-12-17 release. I think we can close this issue. I will check if this issue is specific to 2.0.0b2 release. Thank you all!

0reactions
potiukcommented, Dec 7, 2020

I moved it to 2.0 now - But we need some more info to act on it @vikramarsid - reproducible case.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Python error: execute cannot be used while an asynchronous ...
You are trying to do more than one transaction simultaneously on a single connection. The psycopg documentation says that this is not thread ......
Read more >
execute cannot be used while an asynchronous query is ...
ERROR : execute cannot be used while an asynchronous query is underway · 1. open a query tool for any database in pgadmin...
Read more >
Asynchronous I/O (asyncio) - SQLAlchemy 1.4 Documentation
This ensures that any connections held open by the connection pool will be properly disposed within an awaitable context. Unlike when using ...
Read more >
SQLAlchemy 2.0 - GINO 1.1.0b2 documentation
Simply speaking, SQLAlchemy 1.4 decided to make use of greenlet to mix asynchronous stuff into current code base, avoiding making everything async.
Read more >
Asynchronous I/O (asyncio) - 《SQLAlchemy 1.4 ... - 书栈网
Return True if a transaction is in progress. New in version 1.4.0b2. method sqlalchemy.ext.asyncio.AsyncConnection. in_transaction (). Return ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

DAG serialization JSONDecodeError

Next page

Scheduler keeps exiting with AttributeError: 'MySQLConverter' object has no attribute '_dagruntype_to_mysql' when using docker 2.0.0b2 image