v4b1: RuntimeError: Event loop is closed

I’m not sure whether the root cause is in the 4.0.0 release or in Django-Celery’s 5.3.0 release of two days ago, but since today, we’ve been getting a recurring error in our CI, and after updating the local environment, also locally.

Before, all works well. With these two updates, I get the recurring error shown below.

OS: Windows 10 & Ubuntu 20.04.4 Pip Freeze

alembic==1.8.0
amqp==5.1.1
APScheduler==3.9.1
argparse-addons==0.8.0
asgiref==3.5.2
async-timeout==4.0.2
atomicwrites==1.4.0
attrs==21.4.0
autobahn==22.5.1
Automat==20.2.0
autopage==0.5.1
azure-core==1.24.1
azure-storage-blob==12.12.0
billiard==4.0.0
bitstruct==8.15.1
boto3==1.24.20
botocore==1.27.20
cachetools==5.2.0
cantools==37.1.0
catboost==1.0.6
celery==5.3.0a1
certifi==2022.6.15
cffi==1.15.0
channels==3.0.5
channels-redis==4.0.0b1
charset-normalizer==2.1.0
click==8.1.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
cliff==3.10.1
cloudpickle==2.1.0
cmaes==0.8.2
cmd2==2.4.1
colorama==0.4.5
colorlog==6.6.0
constantly==15.1.0
cryptography==37.0.2
cycler==0.11.0
daphne==3.0.2
defusedxml==0.7.1
Deprecated==1.2.13
diskcache==5.4.0
dj-rest-auth==2.2.4
Django==4.0.5
django-allauth==0.51.0
django-apscheduler==0.6.2
django-celery-beat==2.3.0
django-cors-headers==3.13.0
django-filter==22.1
django-redis==5.2.0
django-timezone-field==5.0
djangorestframework==3.13.1
faiss-cpu==1.7.2
fonttools==4.33.3
google-api-core==2.8.2
google-auth==2.9.0
google-cloud-core==2.3.1
google-cloud-storage==2.4.0
google-crc32c==1.3.0
google-resumable-media==2.3.3
googleapis-common-protos==1.56.3
graphviz==0.20
greenlet==1.1.2
gunicorn==20.1.0
hyperlink==21.0.0
idna==3.3
imbalanced-learn==0.8.0
incremental==21.3.0
iniconfig==1.1.1
isodate==0.6.1
jmespath==1.0.1
joblib==1.1.0
kiwisolver==1.4.3
kombu==5.3.0a1
lightgbm==3.3.2
llvmlite==0.38.1
mailslurp-client==15.11.1
Mako==1.2.0
MarkupSafe==2.1.1
matplotlib==3.5.2
msgpack==1.0.4
msrest==0.7.1
numba==0.55.2
numpy==1.22.4
oauthlib==3.2.0
optuna==2.10.0
packaging==21.3
pandas==1.4.3
patsy==0.5.2
pbr==5.9.0
Pillow==9.1.1
plotly==5.9.0
pluggy==0.13.1
prettytable==3.3.0
prompt-toolkit==3.0.30
protobuf==4.21.2
psycopg2-binary==2.9.3
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
PyJWT==2.4.0
pyOpenSSL==22.0.0
pyparsing==3.0.9
pyperclip==1.8.2
pyreadline3==3.4.1
pytest==6.2.1
pytest-django==4.5.2
python-can==4.0.0
python-crontab==2.6.0
python-dateutil==2.8.2
python3-openid==3.2.0
pytz==2022.1
pytz-deprecation-shim==0.1.0.post0
pywin32==304
PyYAML==6.0
redis==4.3.4
requests==2.28.1
requests-oauthlib==1.3.1
rsa==4.8
s3transfer==0.6.0
scikit-learn==1.0.1
scipy==1.8.1
service-identity==21.1.0
shap==0.40.0
six==1.16.0
slicer==0.0.7
SQLAlchemy==1.4.39
sqlparse==0.4.2
statsmodels==0.13.2
stevedore==3.5.0
tenacity==8.0.1
termcolor==1.1.0
textparser==0.24.0
threadpoolctl==3.1.0
toml==0.10.2
tqdm==4.64.0
Twisted==22.4.0
twisted-iocpsupport==1.0.2
txaio==22.2.1
typing_extensions==4.2.0
tzdata==2022.1
tzlocal==4.2
Unipath==1.1
urllib3==1.26.9
vine==5.0.0
wcwidth==0.2.5
windows-curses==2.3.0
wrapt==1.14.1
xgboost==1.6.1
zope.interface==5.4.0

Traceback:

[2022-06-30 13:37:57,967: ERROR/MainProcess] Event loop is closed
Traceback (most recent call last):
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\connection.py", line 931, in read_response
    response = await self._parser.read_response(
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\connection.py", line 387, in read_response
    raw = await self._buffer.readline()
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\connection.py", line 313, in readline
    await self._read_from_socket()
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\connection.py", line 253, in _read_from_socket
    data = await self._stream.read(self.socket_read_size)
  File "c:\users\anon\appdata\local\programs\python\python39\lib\asyncio\streams.py", line 684, in read
    await self._wait_for_data('read')
  File "c:\users\anon\appdata\local\programs\python\python39\lib\asyncio\streams.py", line 517, in _wait_for_data
    await self._waiter
RuntimeError: Task <Task pending name='Task-4' coro=<AsyncToSync.main_wrap() running at C:\Users\Anon\Project\backend\env\lib\site-packages\asgiref\sync.py:284> cb=[_run_until_complete_cb() at c:\users\anon\appdata\local\programs\python\python39\lib\asyncio\base_events.py:184]> got Future <Future pending> attached to a different loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Anon\Project\backend\app\tasks.py", line 319, in upload
    async_to_sync(channel_layer.group_send)(
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\asgiref\sync.py", line 218, in __call__
    return call_result.result()
  File "c:\users\anon\appdata\local\programs\python\python39\lib\concurrent\futures\_base.py", line 433, in result
    return self.__get_result()
  File "c:\users\anon\appdata\local\programs\python\python39\lib\concurrent\futures\_base.py", line 389, in __get_result
    raise self._exception
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\asgiref\sync.py", line 284, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\channels_redis\core.py", line 570, in group_send
    await connection.zremrangebyscore(
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\client.py", line 487, in execute_command
    return await conn.retry.call_with_retry(
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\retry.py", line 59, in call_with_retry
    return await do()
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\client.py", line 463, in _send_command_parse_response
    return await self.parse_response(conn, command_name, **options)
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\client.py", line 505, in parse_response
    response = await connection.read_response()
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\connection.py", line 943, in read_response
    await self.disconnect()
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\connection.py", line 828, in disconnect
    self._writer.close()  # type: ignore[union-attr]
  File "c:\users\anon\appdata\local\programs\python\python39\lib\asyncio\streams.py", line 353, in close
    return self._transport.close()
  File "c:\users\anon\appdata\local\programs\python\python39\lib\asyncio\selector_events.py", line 700, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "c:\users\anon\appdata\local\programs\python\python39\lib\asyncio\base_events.py", line 746, in call_soon
    self._check_closed()
  File "c:\users\anon\appdata\local\programs\python\python39\lib\asyncio\base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
[2022-06-30 13:38:04,626: ERROR/MainProcess] Event loop is closed
Traceback (most recent call last):
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\connection.py", line 931, in read_response
    response = await self._parser.read_response(
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\connection.py", line 387, in read_response
    raw = await self._buffer.readline()
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\connection.py", line 313, in readline
    await self._read_from_socket()
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\connection.py", line 253, in _read_from_socket
    data = await self._stream.read(self.socket_read_size)
  File "c:\users\anon\appdata\local\programs\python\python39\lib\asyncio\streams.py", line 684, in read
    await self._wait_for_data('read')
  File "c:\users\anon\appdata\local\programs\python\python39\lib\asyncio\streams.py", line 517, in _wait_for_data
    await self._waiter
RuntimeError: Task <Task pending name='Task-10' coro=<AsyncToSync.main_wrap() running at C:\Users\Anon\Project\backend\env\lib\site-packages\asgiref\sync.py:284> cb=[_run_until_complete_cb() at c:\users\anon\appdata\local\programs\python\python39\lib\asyncio\base_events.py:184]> got Future <Future pending> attached to a different loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Anon\Project\backend\app\tasks.py", line 319, in upload
    async_to_sync(channel_layer.group_send)(
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\asgiref\sync.py", line 218, in __call__
    return call_result.result()
  File "c:\users\anon\appdata\local\programs\python\python39\lib\concurrent\futures\_base.py", line 433, in result
    return self.__get_result()
  File "c:\users\anon\appdata\local\programs\python\python39\lib\concurrent\futures\_base.py", line 389, in __get_result
    raise self._exception
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\asgiref\sync.py", line 284, in main_wrap
    result = await self.awaitable(*args, **kwargs)
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\channels_redis\core.py", line 570, in group_send
    await connection.zremrangebyscore(
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\client.py", line 487, in execute_command
    return await conn.retry.call_with_retry(
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\retry.py", line 59, in call_with_retry
    return await do()
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\client.py", line 463, in _send_command_parse_response
    return await self.parse_response(conn, command_name, **options)
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\client.py", line 505, in parse_response
    response = await connection.read_response()
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\connection.py", line 943, in read_response
    await self.disconnect()
  File "C:\Users\Anon\Project\backend\env\lib\site-packages\redis\asyncio\connection.py", line 828, in disconnect
    self._writer.close()  # type: ignore[union-attr]
  File "c:\users\anon\appdata\local\programs\python\python39\lib\asyncio\streams.py", line 353, in close
    return self._transport.close()
  File "c:\users\anon\appdata\local\programs\python\python39\lib\asyncio\selector_events.py", line 700, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "c:\users\anon\appdata\local\programs\python\python39\lib\asyncio\base_events.py", line 746, in call_soon
    self._check_closed()
  File "c:\users\anon\appdata\local\programs\python\python39\lib\asyncio\base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

[bbrowning918]

I did test out the 4.0.0 beta on our staging env and it seemed to work in practice but our test suite blew up. I dug around and think it regresses on this issue:

https://github.com/django/channels/issues/859

The new redis-py connection bits likely can do something like: https://github.com/django/channels_redis/commit/f5e4799e11f472cc267598e9f78099a160f81550

The author of that fix might have some more insight on that as well, I am very new/inexperienced to the async and channels inner workings.

[carltongibson]

@Uninen Super, thank you!

A lot of these errors — Task was destroyed but it is pending! etc. — are expected in the sense that the test aborted so the task didn’t get cleaned up properly. … — That you resolved them is 👍

However, this warning remains no matter what I put in the tests: /proj/venv310/lib/python3.10/site-packages/redis/asyncio/connection.py:677: DeprecationWarning: There is no current event loop

That’s a lingering get_event_loop() presumably. I need to go and look. get_running_loop() is the preferred API.

I’m going to close this. 💃 Looks like we’re green. I will roll out the releases, only a few days late. 😜

Thanks all!