InternalError: Packet sequence number wrong

Hello,

I came across an issue on production that I can’t replicate in the local environment. The same code path often works, but sometimes throws this error. I don’t understand, why this is happening.

I believe there is an issue with multi-thread nature of fastapi and internal usage of pymysql by databases. I’m pretty sure pymysql is single threaded.

But databases was supposed to support asyncio, correct? What am I doing wrong, please?

crud_server.py

db = databases.Database(settings.SQLALCHEMY_DATABASE_URI)

async def get_servers(is_maintenance: bool) -> List[Server]:
    query = ServerTable.select().where(ServerTable.c.is_maintenance == is_maintenance)
    if not db.is_connected:
        await db.connect()
    return await db.fetch_all(query)

ServerTable.py

metadata = MetaData(schema=settings.DBNAME_MAIN)

ServerTable = sqlalchemy.Table(
    "server",
    metadata,
    Column("id", String(15), primary_key=True),
    Column("name", String(100), nullable=False),
    Column("country", String(100), nullable=False),
    Column("group_master", String(15), nullable=True),
    Column("group_slave", String(15), nullable=True, index=True),
    Column("ip", String(length=45), nullable=False),
    Column("max_connections", Integer(), nullable=False),
    Column(
        "is_maintenance",
        Boolean(),
        nullable=False,
        server_default=text("0"),
        default=text("0"),
        index=True,
    ),
    Column("dow_maintenance", String(3), nullable=True),
    Column("hour_maintenance", Integer(), nullable=True),
    Column("minutes_maintenance", Integer(), nullable=True),
    Column("datetime_maintenance", TIMESTAMP(), nullable=True),
)

Error:

  File "/home/admin/tg/app/routers/server_status.py", line 25, in server_status
    servers = await crud_server.get_servers(is_maintenance=False)
  File "/home/admin/tg/app/database/crud_server.py", line 15, in get_servers
    return await db.fetch_all(query)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/databases/core.py", line 140, in fetch_all
    return await connection.fetch_all(query, values)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/databases/core.py", line 239, in fetch_all
    return await self._connection.fetch_all(built_query)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/databases/backends/mysql.py", line 108, in fetch_all
    await cursor.execute(query, args)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/cursors.py", line 239, in execute
    await self._query(query)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/cursors.py", line 457, in _query
    await conn.query(q)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 428, in query
    await self._read_query_result(unbuffered=unbuffered)
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 622, in _read_query_result
    await result.read()
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 1105, in read
    first_packet = await self.connection._read_packet()
  File "/home/admin/.pyenv/versions/3.9.4/envs/venv/lib/python3.9/site-packages/aiomysql/connection.py", line 574, in _read_packet
    raise InternalError(
pymysql.err.InternalError: Packet sequence number wrong - got 0 expected 1

I would really appreciate a hint, what I’m doing wrong. Thank you so much

[frankie567]

I suppose switching to the latest SQLAlchemy 14, which supports asynchio won’t help either, because the underlying driver to MySQL database is buggy, correct?

Yes, unfortunately. SQLALchemy 14 expects you to provide your own asyncio-enabled driver… And aiomysql seems the only option for now.

[houmie]

Hi Frankie,

I no longer see the issue reoccurring. Thanks for your help. I will close the issue and reopen again if it happens again.

In a nutshell two things may have helped.

1. Update of requirements.txt brought in latest FastAPI and SqlAlchemy 1.4, which may have fixed the issue.
2. As you said the conditional check of database connection is creating side effects.

Last but not least the code is much cleaner by using asgi-lifespan. 😃