original errors are swallowed by non-retriable error during savepoint release

UPDATE: simplified the gist to only use sanctioned cockroachdb-python code in the simplest ‘create duplicate’ scenario.

The following code returns sqlalchemy.exc.InternalError because a nested transaction was rolled back. In such a case, I’d like to handle the actual error (eg., a sqlalchemy.IntegrityError due to duplicate insertion) instead.

You can reproduce this with the following code: https://gist.github.com/gpaul/ead175558dddad2b0ea6479f2405bc8c

That yields the following stacktrace showing the actual reason for the failure: sqlalchemy.IntegrityError raised the original exception:

Traceback (most recent call last):
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
psycopg2.IntegrityError: duplicate key value (key)=('some-key') violates unique constraint "configs_key_key"


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 2223, in _flush
    flush_context.execute()
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
    rec.execute(self)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
    uow
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
    mapper, table, insert)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/persistence.py", line 835, in _emit_insert_statements
    execute(statement, params)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
    context)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
    exc_info
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) duplicate key value (key)=('some-key') violates unique constraint "configs_key_key"
 [SQL: 'INSERT INTO configs (key, value) VALUES (%(key)s, %(value)s) RETURNING configs.id'] [parameters: {'key': 'some-key', 'value': 'some-value'}]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
psycopg2.InternalError: SAVEPOINT COCKROACH_RESTART has not been used or a non-retriable error was encountered: current transaction is aborted, commands ignored until end of transaction block


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "repro4.py", line 66, in <module>
    run_transaction(_session_factory, callback)
  File "repro4.py", line 30, in run_transaction
    return ret
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 567, in __exit__
    self.rollback()
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 564, in __exit__
    self.commit()
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 461, in commit
    self._prepare_impl()
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 441, in _prepare_impl
    self.session.flush()
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 2139, in flush
    self._flush(objects)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 2259, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line 76, in __exit__
    compat.reraise(type_, value, traceback)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 2259, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 518, in rollback
    util.reraise(*rollback_err)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 490, in rollback
    transaction._rollback_impl()
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 527, in _rollback_impl
    t[1].rollback()
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1621, in rollback
    self._do_rollback()
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1683, in _do_rollback
    self._savepoint, self._parent)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 750, in _rollback_to_savepoint_impl
    self.engine.dialect.do_rollback_to_savepoint(self, name)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/cockroachdb/sqlalchemy/dialect.py", line 156, in do_rollback_to_savepoint
    connection.execute('ROLLBACK TO SAVEPOINT cockroach_restart')
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 939, in execute
    return self._execute_text(object, multiparams, params)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1097, in _execute_text
    statement, parameters
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
    context)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
    exc_info
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/home/gustav/.pyenv/versions/bouncer/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.InternalError: (psycopg2.InternalError) SAVEPOINT COCKROACH_RESTART has not been used or a non-retriable error was encountered: current transaction is aborted, commands ignored until end of transaction block
 [SQL: 'ROLLBACK TO SAVEPOINT cockroach_restart']

[bdarnell]

I think cockroachdb/cockroach#16846 solves the main problem here, which is that after certain errors the ROLLBACK TO SAVEPOINT would reliably fail.

There’s a secondary issue that if ROLLBACK TO SAVEPOINT fails for other reasons (e.g. network failure), its error clobbers the underlying error that caused the rollback. I don’t think we need to do anything special here - Python 3’s exception chaining already provides all the errors, so nothing is completely hidden, and there’s no special handling that is necessary when these network errors occur (unlike with the original error here, in which the ConditionFailedError might be handled specially).

[gpaul]

Thank you, this was fixed in https://github.com/cockroachdb/cockroach/pull/16846