SQLite ‘no such table: json_each’ error during notification check

Opened from the Prefect Public Slack Community

niko: Hi, we just witnessed an error when running a prefect task locally on a windows 11 machine. This error is not popping up on mac. Do you know how to fix that?

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: json_each
[SQL: INSERT INTO flow_run_notification_queue (flow_run_notification_policy_id, flow_run_state_id) SELECT flow_run_notification_policy.id, CAST(? AS CHAR(36)) AS anon_1
FROM flow_run_notification_policy
WHERE flow_run_notification_policy.is_active IS 1 AND (flow_run_notification_policy.state_names = ? OR EXISTS (SELECT 1
FROM json_each(flow_run_notification_policy.state_names) AS json_each
WHERE json_each.value IN (?))) AND (flow_run_notification_policy.tags = ? OR EXISTS (SELECT 1
FROM json_each(flow_run_notification_policy.tags) AS json_each
WHERE json_each.value IN (SELECT 1 FROM (SELECT 1) WHERE 1!=1)))]
[parameters: ('8d01934d-c9c8-4f23-a3af-ec03ee1586a0', '[]', 'Pending', '[]')]
(Background on this error at: <https://sqlalche.me/e/14/e3q8>)

michael054: <@ULVA73B9P> open “SQLite ‘no such table: json_each’ error during notification check”

michael054: We’ll need to investigate this. Does it happen consistently?

Original thread can be found here.

[sm-Fifteen]

As far as I can tell, the built-in version of SQLite used by Python on Windows, Python\PythonXX\DLLs\sqlite3.dll, doesn’t have the json1 extension (which is where SQLite’s json_each function comes from) statically built-in, which is why I can reproduce the same issue both with aiosqlite and pysqlite/sqlite3.

import aiosqlite
import asyncio
import sqlite3

async def async_main():
    async with aiosqlite.connect(':memory:') as my_db:
        res = await my_db.execute("""SELECT * FROM json_each('["foo", "bar", "baz"]')""")
        rows = await res.fetchall()
        print(rows)

def pysqlite_main():
    with sqlite3.connect(":memory:") as my_db:
        res = my_db.execute("""SELECT * FROM json_each('["foo", "bar", "baz"]')""")
        rows = res.fetchall()
        print(rows)

# asyncio.run(async_main())
pysqlite_main()

Both will throw sqlite3.OperationalError: no such table: json_each because the json_each function is not defined.

Which extensions are statically included at build time is not something SQLite has any way of letting you verify without just sending a query and checking if it fails. Building the json1 extension separately and loading it at runtime (which needs to be done per connection) could be an option, but I’m not sure how well that works with SQLAlchemy.

Another option would be to have Prefect bundle sqlite3.dll and ensure it gets loaded prior to the one included in the Python standard library. That can be done easily without changing the search path by just loading the DLL via ctypes first, since Python won’t reload a library that it sees as “already in memory”.

import ctypes
sqlite3_dll = ctypes.CDLL(r"C:\some\other\path\to\sqlite3.dll")
import sqlite3
import aiosqlite

You can tell by running something like strace python.exe -c 'import ctypes;ctypes.CDLL(r"C:\some\other\path\to\sqlite3.dll");import sqlite3' | grep sqlite.

Combining the two code samples above with the version of sqlite3.dll I happen to have laying around from DB Browser for SQLite fixes the error, both when using aiosqlite and sqlite3 directly.

[madkinsz]

I believe you’ll also have no problems if you use conda since it bundles SQLite with each Python environment.