Failing test when calling django orm code wrapped in database_sync_to_async

I’m trying to test my Channels consumer which calls database_sync_to_async code. The consumer looks something like this:

class MyConsumer(AsyncJsonWebsocketConsumer):
    async def connect(self):
        my_obj = await self.get_obj()
        ...other code

    @database_sync_to_async
    def get_obj(self):
        return MyModel.objects.get(filter_condition)

The test is using the @pytest.mark.asyncio and @pytest.mark.django_db decorators ie:

@pytest.mark.asyncio
@pytest.mark.django_db
async def test_heartbeat():
    communicator = WebsocketCommunicator(MyConsumer, '<path>')
    await communicator.connect()
    await communicator.disconnect()

I’m using the following command to run the test:

./manage.py test xxx/tests.py::test_heartbeat

The test itself passes, however at the end of the test run I always get the following error:

=============================================== ERRORS ===============================================
________________________________ ERROR at teardown of test_heartbeat _________________________________

self = <django.db.backends.utils.CursorWrapper object at 0x7fbc7b8d80b8>, sql = 'DROP DATABASE "test"'
params = None
ignored_wrapper_args = (False, {'connection': <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7fbc7b0481d0>, 'cursor': <django.db.backends.utils.CursorWrapper object at 0x7fbc7b8d80b8>})

    def _execute(self, sql, params, *ignored_wrapper_args):
        self.db.validate_no_broken_transaction()
        with self.db.wrap_database_errors:
            if params is None:
>               return self.cursor.execute(sql)
E               psycopg2.OperationalError: database "test" is being accessed by other users
E               DETAIL:  There is 1 other session using the database.

I can make the test failure go away by removing all references to database_sync_to_async in the consumer, but my understanding is that is poor practice to have sync code (like calling django orm) running inside an async function.

Strangely when I get the failing test two tests run (one pass and one fail), but when I remove the references to database_sync_to_async only one test runs.

Here are the versions of my libraries:

django==2.0.6
daphne==2.2.0
asgiref==2.3.2
channels-redis==2.2.1
pytest==3.6.1
pytest-asyncio==0.8.0
pytest-django==3.1.2

[cybergrind]

I don’t suggest this as a final fix. But we were able to bypass this on django 3.1 + channels 2.4 with such fixture:

@pytest.fixture
def fix_async_db():
    local = connections._connections
    ctx = local._get_context_id()
    for conn in connections.all():
        conn.inc_thread_sharing()
    conn = connections.all()[0]
    old = local._get_context_id
    try:
        with mock.patch.object(conn, 'close'):
            object.__setattr__(local, '_get_context_id', lambda: ctx)
            yield
    finally:
        object.__setattr__(local, '_get_context_id', old)

It is specific for asgi and channels implementation, not threadsafe, not asyncio safe

[tomek-rej]

@vijayshan thanks for your feedback. I did try to use transaction=True in my pytest.mark.django_db, but didn’t seem to help. However I did not try to run your example code exactly, so it’s possible there’s something specific to my database that’s causing those errors.

One thing I noticed was with just a single test case, it passes but as soon as you add more test cases there’s a bigger chance some connections won’t be closed properly resulting in the error message from my original post. Also in case it helps there’s another issue someone created in the pytest-asyncio github that exactly describes my problem: https://github.com/pytest-dev/pytest-asyncio/issues/82. Unfortunately no solution there either.

Anyway in my case I came up with a really hacky fix. It involves calling raw sql to close those connections just before the last test ends. I’m using postgres so my code looks like:

def close_db_connections():
    sql = "select * from pg_stat_activity where datname = 'test'"
    with connection.cursor() as cursor:
        cursor.execute(sql)
        for row in cursor.fetchall()[0:-1]:
            cursor.execute(
                'select pg_terminate_backend({}) from pg_stat_activity'.format(
                    row[2]))

Kind of ugly, I know, but at least now my tests are passing all the time.