500 errors after a few days, giving PendingRollback error

I have a deployment of flask-admin running on a fairly large DB and the interface is used by multiple users.

Every few days, all pages lock up with a 500 error, perhaps because of some race condition where multiple users edit the same thing;

 "Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.9/site-packages/flask_admin/base.py", line 69, in inner
    return self._run_view(f, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flask_admin/base.py", line 368, in _run_view
    return fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flask_admin/model/base.py", line 1969, in index_view
    count, data = self.get_list(view_args.page, sort_column, view_args.sort_desc,
  File "/usr/local/lib/python3.9/site-packages/flask_admin/contrib/sqla/view.py", line 1065, in get_list
    count = count_query.scalar() if count_query else None
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2718, in scalar
    ret = self.one()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2695, in one
    return self._iter().one()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2733, in _iter
    result = self.session.execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1530, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1443, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 310, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1312, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1529, in _execute_context
    conn = self._revalidate_connection()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 508, in _revalidate_connection
    self._invalid_transaction()
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 481, in _invalid_transaction
    raise exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. (Background on this error at: http://sqlalche.me/e/14/8s2b)
"  

It seems that sometimes transactions fail and the session gets stuck in an bad state forever.

I imagine this can be resolved by using a new session for every request and tearing down the sessions at the end, but that might be wasteful and leaky. I also can’t find a good way to set that up, as the flask-admin init already wants the session.

Does anyone have any good ideas on how to fix this? I can’t imagine I’m the only one running into this issue.

[gboor]

Hi @mrjoes,

I was not aware of flask-sqlalchemy doing the whole scoped thing. I switched to flask-sqlalchemy instead of using raw sessions and now everything seems to work fine. Thanks for pointing that one out!

[mrjoes]

Yeah, you were starving your connection pool by not releasing sessions (which, I think, does an explicit rollback on a used connection).