Table Already Exists Error With sqlite + metadata.create_all(checkfirst=True)

Hello - thanks for all the hard work on sqlalchemy!

I have an application where a handful of processes all interact with a single sqlite DB: foo.db. foo.db starts with 0 tables, and each process “lazily” bootstraps foo.db with a table named "some_table".

I notice, however, that when multiple processes call metadata.create_all(checkfirst=True) to try and create "some_table" in foo.db…sqlalchemy winds up raising an OperationalError “table already exists” error.

In other words, running this script…

import concurrent.futures

from sqlalchemy import create_engine, Table, Column, Integer, MetaData


def make_db():
    engine = create_engine("sqlite:///foo.db")
    metadata = MetaData()
    Table(
        "some_table",
        metadata,
        Column("id", Integer, primary_key=True),
    )
    metadata.create_all(bind=engine, checkfirst=True)


def main():
    n = 3
    with concurrent.futures.ProcessPoolExecutor() as executor:
        futures = [executor.submit(make_db) for _ in range(n)]
        for future in concurrent.futures.as_completed(futures):
            future.result()


if __name__ == "__main__":
    main()

… yields the following error:

Traceback (most recent call last):
  File "sqlite_multiprocessing_create.py", line 26, in <module>
    main()
  File "sqlite_multiprocessing_create.py", line 22, in main
    future.result()
  File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 455, in result
    return self.__get_result()
  File "/usr/local/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 414, in __get_result
    raise exception_type, self._exception, self._traceback
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) table some_table already exists
[SQL: 
CREATE TABLE some_table (
	id INTEGER NOT NULL, 
	PRIMARY KEY (id)
)

]
(Background on this error at: http://sqlalche.me/e/e3q8)

Of course, an obvious workaround would be to have, "the parent process bootstrap tables in foo.db. Is it possible, however, to instruct sqlalchemy to use the if not exists clause that sqlite supports?

I’ve read through the notes on sqlite / pysqlite 's DDL semantics, but hadn’t found a nice workaround.

[DavidMuller]

for now I’d either use a @compiles recipe around CreateTable, see https://docs.sqlalchemy.org/en/13/core/compiler.html#synopsis

This is a @compiles recipe I came up with to inject IF NOT EXISTS into sqlite CREATE TABLE DDL statements (based on @zzzeek 's above pointer):

import re

from sqlalchemy.ext.compiler import compiles
from sqlalchemy.schema import CreateTable
from sqlalchemy import create_engine, Table, Column, Integer, MetaData


class CreateTableIfNotExists(CreateTable):
    pass


@compiles(CreateTableIfNotExists)
def compile_create_table_if_not_exists(element, ddlcompiler, **kw):
    default_create_ddl = ddlcompiler.visit_create_table(element, **kw)
    default_create_ddl = default_create_ddl.lstrip()
    create_if_not_exists = re.sub(
        r"^CREATE TABLE ", "CREATE TABLE IF NOT EXISTS ", default_create_ddl, flags=re.IGNORECASE
    )
    return create_if_not_exists


engine = create_engine("sqlite:///" + "foo.db")
metadata = MetaData()
table = Table(
    "some_table",
    metadata,
    Column("id", Integer, primary_key=True),
)
engine.execute(CreateTableIfNotExists(table))

Regex is obviously not the greatest, but wanted to share the code in case its useful to others as an example

(Can move this comment over to #2843 too @zzzeek , but didn’t want to sidetrack that issue)

[zzzeek]

yes regex is kind of how it goes for the recipes right now, I think this is fine.