Getting sqlalchemy.exc.IntegrityError when executing simultaneous inserts.

Hi,

Thank you so much for creating SQLAlchemy!

Summary

We are currently seeing the following error:

sqlalchemy.exc.IntegrityError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(mysql.connector.errors.IntegrityError) 1062 (23000): Duplicate entry '1-62' for key 'user_id_item_id'
[SQL: INSERT INTO item_user_associations (user_id, item_id, extra_data) VALUES (%(user_id)s, %(item_id)s, %(extra_data)s)]
[parameters: {'user_id': 1, 'item_id': 62, 'extra_data': 'hello world!'}]
(Background on this error at: http://sqlalche.me/e/gkpj)

We believe the error occurs when users send two POST requests to our API containing similar data in very close succession. However, the issue is reproducible without networking.

Here is an example PyCharm project to reproduce the issue: AssociationErrorExample-test.zip

Steps to Repro

1. Get a local copy of MySQL running:

    docker run --name my_db \
         -e MYSQL_DATABASE=my_db \
         -e MYSQL_USER=user \
         -e MYSQL_PASSWORD=password \
         -p 3306:3306 -d mysql/mysql-server:5.7
   

2. Run the “Compound” run configuration from PyCharm:

   Run > Run 'Compound'
   

   This executes the “example” run configuration 8 times in parallel.
3. As an alternative you can execute example.py in multiple terminal windows.

Expected: No error.

Actual: You see an sqlalchemy.exc.IntegrityError error in at least one of the outputs.

Workaround

We’ve been doing something like this to overcome this issue:

@contextmanager
def session_scope():
    """
    This code is taken from the final example here:
    https://docs.sqlalchemy.org/en/13/orm/session_basics.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it
    """
    session = Session()
    try:
        yield session
        session.commit()
    except exc.InternalError:
        session.rollback()
        session.commit()
    except:
        session.rollback()
        raise
    finally:
        session.close()

Questions

Is there a built in SQLAlchemy mechanism or recommended best practice to overcome this issue?

Please let me know. Thanks!

-Nick

[advance512]

@zzzeek Are there plans to support “upsert” operations for MySQL (and other databases that support it, e.g. sqlite) in the ORM, as opposed to Core?

I have spent hours and found no simple way to add support to a single ORM model to do “upserts” instead of normal inserts. Everything requires dropping down to Core, or performing global changes to all models.

[zzzeek]

added this at #5441