Not Handling Connections to Multiple Worker Processes Correctly

if you just change the example qnotify.py script (or indeed any other) to run on two worker processes as follows:

jp.justpy(notify_test, start_server=False)
app = jp.app    

if __name__ == '__main__':
    import uvicorn
    uvicorn.run('qnotify:app',host='127.0.0.1',port=80,workers=2,use_colors=False)

keep refreshing the page and hitting one of the arrow buttons. Eventually the buttons will stop working and you will get the error message on every button click: WARNING justpy: No page to load After a Ctrl+C in the stack trace you will see:

  File "c:\users\dima\anaconda3\lib\site-packages\justpy\justpy.py", line 279, in on_disconnect
    await WebPage.instances[pid].on_disconnect(websocket)   # Run the specific page disconnect function
KeyError: 0

I believe this stack trace actually points to the issue where the page stops working, even though you don’t normally get to see it when it occurs. I’ve only come to this conclusion after running the script via pm2, which opens separate command line windows for each forked worker process.

Whenever the workers load balance a page load, so that the page http request goes to one worker and the websocket connection to the other (easy to see in separate shell windows), that’s when the exception occurs and the page stops working.

Maybe it’s an implementation issue with Uvicorn, but it is justpy that’s throwing the exception.

A refresh will usually still get the page back.

Btw, the KeyError continues to increment if the same issue re-occurs.

[platinops]

This was very helpful, much appreciated!

Only tested it with two identical app instances running on different ports, and works like a charm. Both sections you mentioned were indeed required for websockets to work.

My config below (on nginx-1.21.6). Note this is running on an internal network, not facing the www.

worker_processes  1;

events {
    worker_connections  1024;
}

http {

	# To prevent leaking info on your server and system - test on https://gf.dev/server-signature-test
	server_tokens off;

	# For websockets
	map $http_upgrade $connection_upgrade {
		default upgrade;
		'' close;
	}

	# The end-points to load balance to
	upstream webapp {
		
		# Make sure that requests from each IP are handled by the same worker
		ip_hash;
		
		# This directive is not compatible with ip_hash and messes up websockets => do not use
		# least_conn; 
		
		# Workers - each one running a single Uvicorn>Starlette>Justpy app instance 
		# Avoid using "localhost" here
		server 127.0.0.1:10001; 
		server 127.0.0.1:10002;
	}

    server {

		listen 443 http2 ssl;
		listen [::]:443 http2 ssl;
		
		location / {
		
			# Hook into the back-end servers
			proxy_pass https://webapp; # note https
			proxy_redirect off;
			proxy_buffering off;
			proxy_connect_timeout 10s;
			client_max_body_size 20M; # upload size limit
			
			# This section is for websockets
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection $connection_upgrade;
			proxy_set_header Host $host;

		}
		
		# SSL config
		
		ssl_certificate      ssl/my_site.crt;
		ssl_certificate_key  ssl/my_site.key;
		
		# Disallow 1 and 1.1, re TLS security see https://www.packetlabs.net/tls-1-1-no-longer-secure/
		ssl_protocols TLSv1.3 TLSv1.2;
		
		ssl_prefer_server_ciphers on;
		ssl_session_timeout 1d;
		ssl_session_cache shared:SSL:50m;
		ssl_session_tickets off;   # Requires nginx >= 1.5.9
		ssl_stapling on;           # Requires nginx >= 1.3.7
		ssl_stapling_verify on;    # Requires nginx => 1.3.7  
		add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload";
		
		# Use ONLY the strongest ciphers! from https://syslink.pl/cipherlist/
		ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
		ssl_ecdh_curve secp384r1;  # Requires nginx >= 1.1.0
		
    }
	
}

[BoogeyOff]

No problems. Below is a minimal example of the config you’d need for nginx. Then you’d run JustPy instances on ports 10001 and 10002 over https. You can give it the same key and crt files as for nginx.

The bulk of the SSL config probably isn’t “necessary”, but you might as well follow what according to my research is close to best practice. I’ve omitted a lot of other things I mentioned above, eg nginx serving static files - there’s about a million config settings for nginx and it would just confuse the issue to post a full config here.

Feel free to play around with this. My memory is a bit hazy on what some of the settings do exactly.

events {
	worker_connections 1024;
}

http {

        # to prevent leaking info on your server and system - test on https://gf.dev/server-signature-test
        server_tokens off;

	# think this is for web sockets (can't remember)
	map $http_upgrade $connection_upgrade {
		default upgrade;
		'' close;
	}
	
	# the end-points for dynamic content servers to load balance to
	upstream webapp_dyn {
		ip_hash;
		server 127.0.0.1:10001 weight=1; # avoid using "localhost" here
		server 127.0.0.1:10002 weight=1;
	}

	# the actual https server config (redirects to backend to according to webapp_dyn config above)
	server {
	
		listen 443 http2 ssl;
		listen [::]:443 http2 ssl;

		server_name your.web.domain;
		
		location / {
			# hook into the back-end servers
			proxy_pass https://webapp_dyn; # note https
			proxy_redirect off;
			proxy_buffering off;
			proxy_connect_timeout 10s;
			client_max_body_size 20M; # upload size limit
			
			# turn off the cache for the dynamic content
			proxy_no_cache 1;
			# even if cached, don't try to use it
			proxy_cache_bypass 1; 
			# another way to kill the cache
			expires -1;
			
			# this section is for websockets I think
			proxy_http_version 1.1;
			proxy_set_header Upgrade $http_upgrade;
			proxy_set_header Connection $connection_upgrade;
			proxy_set_header Host $host;


		}

		# --------------------- the ssl security config -----------------------------------
		
		# give it the site certificate and key
		# note: with proper certificate chain, just give it one bundle file that's a cat of the authorities and the site
		# generate via: cat your_domain.crt intermediate.crt root.crt >> ssl-bundle.crt
		# see https://www.ssls.com/knowledgebase/how-to-install-an-ssl-certificate-on-a-nginx-server/
		
		ssl_certificate your_site.crt;
		ssl_certificate_key your_site.key;

		# disallow 1 and 1.1, re TLS security see https://www.packetlabs.net/tls-1-1-no-longer-secure/
		ssl_protocols TLSv1.3 TLSv1.2;
		
		ssl_prefer_server_ciphers on;
		ssl_session_timeout 1d;
		ssl_session_cache shared:SSL:50m;
		ssl_session_tickets off; # Requires nginx >= 1.5.9
		ssl_stapling on; # Requires nginx >= 1.3.7
		ssl_stapling_verify on; # Requires nginx => 1.3.7  
		add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload";
		
		# use ONLY the strongest ciphers! from https://syslink.pl/cipherlist/
		ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
		ssl_ecdh_curve secp384r1; 				# Requires nginx >= 1.1.0

		# diffie hellman key exchange params
		ssl_dhparam .\dhparam.pem; # generate file via: openssl dhparam -out .\dhparam.pem 4096

	}

}



#