Sessions data corrupted

• Ubuntu 16.04
• Channels 2.1.1, Django 2.0.4
• Served background tasks via daphne, with nginx as a web server

Traceback:

May 15 20:30:52 ip-172-21-33-32 daphne[9222]: WARNING 2018-05-15 20:30:52,086 Session data corrupted May 15 20:30:52 ip-172-21-33-32 daphne[9222]: DEBUG 2018-05-15 20:30:52,088 HTTP 403 response started for [‘127.0.0.1’, 48324] May 15 20:30:52 ip-172-21-33-32 daphne[9222]: DEBUG 2018-05-15 20:30:52,089 HTTP close for [‘127.0.0.1’, 48324] May 15 20:30:52 ip-172-21-33-32 daphne[9222]: DEBUG 2018-05-15 20:30:52,090 HTTP response complete for [‘127.0.0.1’, 48324]

Sessions served by uwsgi are correctly being served.

This is how is served by nginx:

server {
    listen 443;
    ssl on;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Disable SSLv3 (CVE-2014-3566)

    # Disable emitting nginx version in error messages and in the "Server"
    # response header field.
    server_tokens off;

    ssl_certificate       project.crt;
    ssl_certificate_key    project.key;
    ssl_client_certificate project.ca;
    ssl_verify_client      optional;

    # Set the maximum allowed size of the client request body, specified in the
    # "Content-Length" request header field.
    client_max_body_size 20M;

    # These requests are handled by daphne
    location /stream {
        # daphne server running on port 8001 so we set a proxy to that url
        proxy_pass http://0.0.0.0:8001;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_redirect   off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $server_name;
    }

    # These requests are handled by uwsgi
    location / {
        include uwsgi_params;
        uwsgi_pass unix:/run/uwsgi/app/project/socket;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        # Make the following two variables accessible to the application:
        uwsgi_param SSL_CLIENT_VERIFY $ssl_client_verify;
        uwsgi_param SSL_CLIENT_RAW_CERT $ssl_client_raw_cert;
    }
}

[dlernstrom]

Just as a follow up, in case anybody ever reads through this thread and discovers the Session Data Corrupted issue that I was being faced with, I’ll add an update to what we were faced with. As I mentioned above, allowing Daphne to serve the traditional request/response stuff and disabling Gunicorn did completely resolve my Session Data Corrupted errors.

As it turns out, the error that I’ve been plagued with now has to do with the combination of the websocket timeout as well as the group_expiry (24 hours, https://github.com/django/channels/issues/999). In my infrastructure, I have a fleet of RPi devices that connect in to Daphne/Channels over a web socket so that the server can push out information. These devices create a custom channel unique to the deployment scenario of the device. What I’m discovering is that Channels expires the group and the RPi device doesn’t find out the group was destroyed because the web socket session is still active. I realize that my issue now diverges from the topic of this thread, so I’m going to find a different venue to pose the question of how to get the group_expiry to play nicely - probably Stack Overflow. Cheers!

[dlernstrom]

Hey @andrewgodwin

So, I realize I hadn’t updated in a couple of months, but as it turns out, this problem is still affecting me… at least it was (perhaps hopefully fixed now). I’m posting this for informational sake, not necessarily to reopen this ticket, but to document what I am experiencing in order to (hopefully) alleviate somebody else that may be experiencing the same thing. I’m extremely new to websockets, Redis, and items like reverse proxy through Nginx and such.

Essentially, what I’m finding is that if I allow Daphne/Channels to run for more than about a couple of days without restarting the Daphne process, then I’ll eventually experience failures, websockets no longer work, and the log shows attempts for websocket connection, but then also a line that says Session data corrupted.

I deploy code with Ansible, which includes a Daphne restart at the tail end. Since this project has been in active development for me, it is pretty rare that a couple of days would go past without me deploying code, though, so it hasn’t happened that frequently (admittedly, I haven’t really studied intently the log data on the server for log info that I did not myself emit).

Now, here’s the interesting bit. Since introduction of websockets is a late addition to the project, and it is fairly new tech for us, we proxied only the websocket portion of the Django app, but retained the Gunicorn for the traditional request/response handling. We used the same project, though, with the same Django configuration files and such for both legs of the application.

Here’s where I start to hypothesize a bit. I suspect that under the covers, Daphne and Gunicorn are doing something different with session headers, session handling, session expiration, or it might even be that my proxy settings are slightly different in my Nginx stack for the Daphne leg and the Gunicorn leg. What I think is occurring, though, is that Gunicorn will create some session data, but when Daphne touches it, it seems foreign, so django.security.SuspiciousSession base.py line 115 of the Django code logs Session Data Corrupted, and forces a reconnect and such.

Tonight, I had the idea of elbowing Gunicorn out of the picture entirely and letting Daphne handle both the persistent websocket, but also the short-lived request/response. My Session Data Corrupted messages have stopped occurring in the logs (which is a great sign).

Ultimately, I will let this thing ride a few days to see if I get any relief from the problems I’ve been experiencing. Please let me know if I’m off base or if I should be targeting a different avenue of exploration.

As I’ve shared in the past, thank you for your continued hard work. It is extremely appreciated!

Cheers, David

Oh, and for reference: channels==2.1.3 channels_redis==2.2.1 Django==2.1 Python 3.6.6