question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Websockets allow_origin can't be overridden, leading to 403 errors using dask-labextension behind reverse proxies

See original GitHub issue

Bug description

Websockets allow_origin can’t be overridden, leading to 403 errors using dask-labextension behind reverse proxies

Expected behaviour

We are trying to use dask-labextension in proxy mode (ie, putting /proxy/8787 in the labextension search bar) within Open OnDemand. Labextension tabs should display graphs etc (and do in the case we use ssh portforwarding instead of the ondemand interface).

Actual behaviour

Labextension opens tabs in the workspace, but they remain blank and the browser console shows 403 errors opening websockets

How to reproduce

See https://github.com/dask/dask-labextension/issues/176 - basically start a dask cluster, point the labextension at it and attempt to open one of the tabs (within Open OnDemand, or some other reverse proxy system where the Host header won’t match the origin).

Your personal set up

We are running jupyterlab instances through Open OnDemand (https://openondemand.org/), using a pip installed venv on Centos 8. OnDemand generates a config.py which includes c.NotebookApp.allow_origin = ‘*’ and c.NotebookApp.disable_check_xsrf = True

I’ve dug into the code quite a bit, and as far as I can tell the problem is that jupyter_server_proxy/websocket.py doesn’t override the check_origin WebSocketHandler method in the WebSocketHandlerMixin class. As a result, it falls back to the method in the base tornado class, which strictly checks that the Host header and the origin match. Because of the OnDemand reverse proxy setup we are using that won’t be the case.

For non-websocket traffic I think this ultimately gets handled by notebook/base/handlers.py IPythonHandler class (or possibly jupyter_server/base/handlers.py JupyterHandler), which overrides check_origin and simply returns True if allow_origin is set to ‘*’.

When I override the check_origin method in WebSocketHandlerMixin to always return True, the labextension works as expected. I expect that just cloning the code from notebook/base/handlers.py would work fine, but I’ve no idea if that would open up nasty security issues or something.

Full environment pip freeze output (python 3.7.4 venv):
aiohttp==3.7.3
anyio==2.1.0
argon2-cffi==20.1.0
async-generator==1.10
async-timeout==3.0.1
attrs==20.3.0
Babel==2.9.0
backcall==0.2.0
bleach==3.3.0
bokeh==2.2.3
certifi==2020.12.5
cffi==1.14.5
chardet==3.0.4
click==7.1.2
cloudpickle==1.6.0
dask==2021.2.0
dask-jobqueue==0.7.2
dask-labextension==5.0.0
decorator==4.4.2
defusedxml==0.6.0
distributed==2021.2.0
entrypoints==0.3
HeapDict==1.0.1
idna==2.10
importlib-metadata==3.4.0
ipykernel==5.4.3
ipython==7.20.0
ipython-genutils==0.2.0
jedi==0.18.0
Jinja2==2.11.3
json5==0.9.5
jsonschema==3.2.0
jupyter-client==6.1.11
jupyter-core==4.7.1
jupyter-server==1.3.0
jupyter-server-proxy==1.6.0
jupyterlab==3.0.7
jupyterlab-pygments==0.1.2
jupyterlab-server==2.2.0
MarkupSafe==1.1.1
mistune==0.8.4
msgpack==1.0.2
multidict==5.1.0
nbclassic==0.2.6
nbclient==0.5.2
nbconvert==6.0.7
nbformat==5.1.2
nest-asyncio==1.5.1
notebook==6.2.0
numpy==1.20.1
packaging==20.9
pandocfilters==1.4.3
parso==0.8.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.1.0
prometheus-client==0.9.0
prompt-toolkit==3.0.16
psutil==5.8.0
ptyprocess==0.7.0
pycparser==2.20
Pygments==2.8.0
pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
pytz==2021.1
PyYAML==5.4.1
pyzmq==22.0.3
requests==2.25.1
Send2Trash==1.5.0
simpervisor==0.4
six==1.15.0
sniffio==1.2.0
sortedcontainers==2.3.0
tblib==1.7.0
terminado==0.9.2
testpath==0.4.4
toolz==0.11.1
tornado==6.1
traitlets==5.0.5
typing-extensions==3.7.4.3
urllib3==1.26.3
wcwidth==0.2.5
webencodings==0.5.1
yarl==1.6.3
zict==2.0.0
zipp==3.4.0
Configuration config.py:
c.NotebookApp.ip = '*'
c.NotebookApp.port = 24778
c.NotebookApp.port_retries = 0
c.NotebookApp.password = u'sha1:ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ'
c.NotebookApp.base_url = '/node/XXXX/24778/'
c.NotebookApp.open_browser = False
c.NotebookApp.allow_origin = '*'
c.NotebookApp.notebook_dir = '/home/YYYYY'
Logs
[W 2021-02-12 18:12:54.610 ServerApp] 403 GET /node/XXXX/58092/proxy/8787/individual-task-stream/ws (10.1.9.164) 2.05ms referer=None

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
davidmonrocommented, Nov 30, 2021

I can confirm it is now fixed.

0reactions
yuvipandacommented, Nov 30, 2021

I think this was fixed in #295

Read more comments on GitHub >

github_iconTop Results From Across the Web

Minio returning 403 when using a reverse proxy #7736 - GitHub
I'm using Nginx as reverse proxy, then CloudFlare on top of it: ... in Minio server when you run the mc command that...
Read more >
Nginx Reverse Proxy Websocket Authentication - HTTP 403
The connection always fails with a HTTP 403 code. I'm using version 1.9.7. Do you have any idea, why the client does not...
Read more >
403 Forbidden when trying to access a website using Proxy
Hi @Nanda Malve​ ,. I tried to check for logs related to reference error, but am unable to parse the reference string.
Read more >
Controller v5.11 behind reverse proxy gives 403 on login
Hi, I've been running the unifi controller locally behind an Apache reverse proxy for-nearly-ever, but after the last upgrade (I think from 5.10...
Read more >
How to Fix a 403 Forbidden Error on Your WordPress Site
To help you fix the 403 Forbidden Error on your WordPress site, we'll cover five separate troubleshooting steps in detail: File permissions ....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found