Channel Layer doesn't work for AsyncHttpConsumer

• Your OS and runtime environment, and browser if applicable MacOS 10.13.6 x64, Python 3.5.3, Django version 2.0.5, ASGI/Channels version 2.1.6
• A pip freeze output showing your package versions

absl-py==0.2.2
astor==0.6.2
astroid==1.6.4
bleach==1.5.0
CacheControl==0.12.4
cachetools==2.1.0
certifi==2018.4.16
channels==2.1.6
chardet==3.0.4
click==6.7
decorator==4.3.0
dj-database-url==0.5.0
Django==2.0.5
djangorestframework==3.8.2
firebase-admin==2.10.0
future==0.16.0
gast==0.2.0
gcloud==0.17.0
geocoder==1.38.1
google-api-core==1.2.0
google-auth==1.4.1
google-cloud-core==0.28.1
google-cloud-firestore==0.29.0
google-cloud-storage==1.10.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
googlemaps==2.5.1
grpcio==1.12.0
gunicorn==19.8.1
html5lib==0.9999999
httplib2==0.11.3
idna==2.6
isort==4.3.4
jws==0.1.3
lazy-object-proxy==1.3.1
Markdown==2.6.11
mccabe==0.6.1
msgpack-python==0.5.6
numpy==1.14.3
oauth2client==3.0.0
protobuf==3.5.2.post1
pyasn1==0.4.3
pyasn1-modules==0.2.1
pycryptodome==3.4.3
pylint==1.9.1
python-jwt==2.0.1
pytz==2018.4
ratelim==0.1.6
requests==2.11.1
requests-toolbelt==0.7.0
reverse-geocoder==1.5.1
rsa==3.4.2
scipy==1.1.0
six==1.11.0
tensorboard==1.8.0
tensorflow==1.8.0
termcolor==1.1.0
urllib3==1.22
Werkzeug==0.14.1
whitenoise==3.3.1
wrapt==1.10.11
Pyrebase==3.0.27
psycopg2
django-cors-headers

• What you expected to happen vs. what actually happened This is my LongPollingConsumer where inside handle I am storing the channel_name in a database then doing the usual as documented here.

class LongPollConsumer(AsyncHttpConsumer):
  async def handle(self, body):
    self.room_name = self.scope['url_route']['kwargs']['uuid']
    # self.room_group_name = 'upvote_'
    # await self.send_response(200, b"Hello", headers=[
    #     (b"Content-Type", b"application/json"),
    # ])
    store_channel(self.room_name, self.channel_name)
    await self.send_headers(status=200, headers=[
        (b"Content-Type", b"application/json"),
    ])
    print('consumer room_name', self.room_group_name)
    # await self.channel_layer.group_add(
    #     self.room_group_name,
    #     self.channel_name
    # )

    # Headers are only sent after the first body event.
    # Set "more_body" to tell the interface server to not
    # finish the response yet:
    await self.send_body(body=b"Response", more_body=True)

  async def upvote_message(self, event):
    delete_channel(self.room_name)
    print(event['message'])
    # Send JSON and finish the response:
    # async_to_sync(self.channel_layer.group_discard)(
    #     self.room_group_name,
    #     self.channel_name
    # )
    await self.send_body(body=json.dumps(event).encode("utf-8"))

handle is invoked as it should be. Then from my views.py, I am retrieving list of channels from the database and dispatching the handler(upvote_message) with an event message

channel_layer = get_channel_layer()
channels = retrieve_channels(uuid)
for channel in channels.keys():
    print('dispatching to channel', channel)
    async_to_sync(channel_layer.send)(
    channel, {
            "type": "upvote.message",
            "message": {
                'actionType': 'UPVOTES_LONG_POLL_RESPONSE',
                'data': {}
            }
        }
    )

Expected behaviour is that for every channel_layer.send, the async method upvote_message inside my LongPollingConsumer should be invoked with appropriate event message and close the respective HTTP connection. But nothing is happening. From views.py, it prints

sending to channel specific..inmemory!KTTXyBUZnIwj
sending to channel specific..inmemory!bafQMBaSCOmu
sending to channel specific..inmemory!zAGgrVeMWNth
sending to channel specific..inmemory!tazNTCBhKvfh
sending to channel specific..inmemory!XuMVhdIjBCAi

and the curl (using curl -X GET http://localhost:8000/lp/upvote/-L_v3R0Xp-BlOeagBuu1/ --verbose) remains stuck at:

*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 8000 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8000 (#0)
> GET /lp/upvote/-L_v3R0Xp-BlOeagBuu1/ HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Transfer-Encoding: chunked
<

then eventually after manually closing connection,

* transfer closed with outstanding read data remaining
* stopped the pause stream!
* Closing connection 0

• How you’re running Channels (runserver? daphne/runworker? Nginx/Apache in front?) python manage.py runserver development server
• Console logs and full tracebacks of any errors Absolutely no logs generated except and the print statements in the code above.

Django version 2.0.5, using settings 'AppName.settings'
Starting ASGI/Channels version 2.1.6 development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

[malefice]

So, I did a little more digging into the channels source code, and my investigation led me to this and this.

The await_many_dispatch method is the one responsible for dispatching stuff like http.request, websocket.connect, and all other custom “events” sent. For AsyncHttpConsumer, the http.request event will be dispatched to the http_request method.

According to the source code, the http_request method will always raise a StopConsumer exception once the handle method has finished. This exception will, in turn, trigger the finally clause in await_many_dispatch which will cancel all tasks, including the task handling channel layer events.

Putting in a while loop inside the handle method will also not work, because it is called from inside the task handling http.request, and if that task does not end, the task handling channel layer events will not be given a chance to run.

What I did to validate my findings was to use a keepalive flag that will be turned on if send_body was invoked with more_body=True, and if that flag is on, the StopConsumer exception will not be raised. With the example code below, I can get the chat_message method to trigger.

class SseConsumer(AsyncHttpConsumer):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.keepalive = False

    async def handle(self, body):
        print('start handle')
        await self.send_headers(headers=[
            (b'Cache-Control', b'no-cache'),
            (b'Content-Type', b'text/event-stream'),
            (b"Transfer-Encoding", b"chunked"),
            (b'Access-Control-Allow-Origin', b'*'),
        ])

        await self.send_body(b'', more_body=True)
        await self.channel_layer.group_add('test123', self.channel_name)

    async def send_body(self, body, *, more_body=False):
        if more_body:
            self.keepalive = True
        assert isinstance(body, bytes), "Body is not bytes"
        await self.send(
            {"type": "http.response.body", "body": body, "more_body": more_body}
        )

    async def http_request(self, message):
        if "body" in message:
            self.body.append(message["body"])
        if not message.get("more_body"):
            try:
                await self.handle(b"".join(self.body))
            finally:
                if not self.keepalive:
                    await self.disconnect()
                    raise StopConsumer()

    async def chat_message(self, event):
        payload = 'event: test\ndata: 2\n\n'
        await self.send_body(payload.encode('utf-8'), more_body=True)

I am not sure what other stuff this might break, but I can submit a PR if this is adequate, otherwise I will defer to the masters. At the very least, part of the investigation has already been performed, and hopefully this helped save some time.

[rohit20001221]

class SseConsumer(AsyncHttpConsumer):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.keepalive = False

    async def handle(self, body):
        print('start handle')
        await self.send_headers(headers=[
            (b'Cache-Control', b'no-cache'),
            (b'Content-Type', b'text/event-stream'),
            (b"Transfer-Encoding", b"chunked"),
            (b'Access-Control-Allow-Origin', b'*'),
        ])

        await self.send_body(b'', more_body=True)
        await self.channel_layer.group_add(self.group_name, self.channel_name)

    async def send_body(self, body, *, more_body=False):
        if more_body:
            self.keepalive = True
        assert isinstance(body, bytes), "Body is not bytes"
        await self.send(
            {"type": "http.response.body", "body": body, "more_body": more_body}
        )

    async def http_request(self, message):
        if "body" in message:
            self.body.append(message["body"])
        if not message.get("more_body"):
            try:
                await self.handle(b"".join(self.body))
            finally:
                if not self.keepalive:
                    await self.disconnect()
                    raise StopConsumer()



modified the SseConsumer to Generalize it you can see the below example for reference


class ExampleNotifier(SseConsumer):
    group_name = 'example_group'

    async def handle_user_data(self, event):
        payload = 'data: %s\n\n' % json.dumps(event["message"])

        await self.send_body(payload.encode('utf-8'), more_body=True)




await layer.group_send('example_group', {'message': {'user': 'krishna'}, "type":"handle.user.data"})