Django 3 and database_sync_to_async

I am having an issue with Django 3 and channels 2.4.0.

when I call database_sync_to_async I get this

Error
Traceback (most recent call last):
  File "/usr/lib/python3.7/unittest/mock.py", line 1255, in patched
    return func(*args, **keywargs)
  File "/home/waqas/PycharmProjects/nethub/apps/apis/tests/test_models/test_models.py", line 98, in test_process_module
    output = async_to_sync(self.module_1.process_module)(output_fetcher, requested_data)
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/asgiref/sync.py", line 116, in __call__
    return call_result.result()
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/asgiref/sync.py", line 156, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "/home/waqas/PycharmProjects/nethub/apps/apis/models.py", line 135, in process_module
    required_keys = {item.key_name: item.text_fms_field_name for item in keys_data}
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/db/models/query.py", line 276, in __iter__
    self._fetch_all()
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/db/models/query.py", line 1261, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/db/models/query.py", line 57, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
    cursor = self.connection.cursor()
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/utils/asyncio.py", line 24, in inner
    raise SynchronousOnlyOperation(message)
django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.

If is set

os.environ["DJANGO_ALLOW_ASYNC_UNSAFE"] = "true"

I don’t get the above error but then I get this after few seconds. Looks like thread keeps running.

Destroying test database for alias 'default'...
Traceback (most recent call last):
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql)
psycopg2.errors.ObjectInUse: database "test_nethub" is being accessed by other users
DETAIL:  There is 1 other session using the database.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/waqas/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/193.6015.41/plugins/python/helpers/pycharm/django_test_manage.py", line 168, in <module>
    utility.execute()
  File "/home/waqas/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/193.6015.41/plugins/python/helpers/pycharm/django_test_manage.py", line 142, in execute
    _create_command().run_from_argv(self.argv)
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/core/management/commands/test.py", line 23, in run_from_argv
    super().run_from_argv(argv)
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/home/waqas/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/193.6015.41/plugins/python/helpers/pycharm/django_test_manage.py", line 104, in handle
    failures = TestRunner(test_labels, **options)
  File "/home/waqas/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/193.6015.41/plugins/python/helpers/pycharm/django_test_runner.py", line 255, in run_tests
    extra_tests=extra_tests, **options)
  File "/home/waqas/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/193.6015.41/plugins/python/helpers/pycharm/django_test_runner.py", line 156, in run_tests
    return super(DjangoTeamcityTestRunner, self).run_tests(test_labels, extra_tests, **kwargs)
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/test/runner.py", line 694, in run_tests
    self.teardown_databases(old_config)
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/test/runner.py", line 638, in teardown_databases
    keepdb=self.keepdb,
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/test/utils.py", line 297, in teardown_databases
    connection.creation.destroy_test_db(old_name, verbosity, keepdb)
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/db/backends/base/creation.py", line 259, in destroy_test_db
    self._destroy_test_db(test_database_name, verbosity)
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/db/backends/base/creation.py", line 276, in _destroy_test_db
    % self.connection.ops.quote_name(test_database_name))
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/waqas/.venv/nethub/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql)
django.db.utils.OperationalError: database "test_nethub" is being accessed by other users
DETAIL:  There is 1 other session using the database.

[carltongibson]

This line:

    required_keys = {item.key_name: item.text_fms_field_name for item in keys_data}

Looks like you’re walking object attributes, causing ORM look-ups, which aren’t allowed in async contexts. You need to prefetch all the required relationships, or put this into a sync_to_async wrapped function.

[tehfink]

@waqasraz I had similar problems with database_sync_to_async, and eventually tracked them down to attributes or references to m2m called from the function wrapped with database_sync_to_async.