[AzureAD, GitLab, maybe more] Use of web traffic proxy cause doesn't work

Actual behaviour

Our VMs with jupyterhub and oauth installed are behind a proxy. Jupyter comes up fine but fails to login azure-active directory user with the proxy set.

i.e when proxy is set in the shell before starting the jupyter as follows

http_proxy=http://x.x.x.x:xxxx
https_proxy=http://x.x.x.x:xxxx
no_proxy="xxx"

jupyter comes up fine with aadlogin screen, however it fails to login the user with the following error

See error logs

[D 2021-01-21 16:55:32.169 JupyterHub log:181] 200 GET /hub/logo (@::ffff:10.250.6.187) 0.74ms
[D 2021-01-21 16:55:32.236 JupyterHub log:181] 200 GET /hub/static/favicon.ico?v=fde5757cd3892b979919d3b1faa88a410f28829feb5ba22b6cf069f2c6c98675fceef90f932e49b510e74d65c681d5846b943e7f7cc1b41867422f0481085c1f (@::ffff:10.250.6.187) 0.56ms
[I 2021-01-21 16:55:33.537 JupyterHub oauth2:104] OAuth redirect: 'https://vm-dsvmv2-ihprodtest2.bdaa.prod.ams.azu.dbgcloud.io:8000/hub/oauth_callback'
[D 2021-01-21 16:55:33.538 JupyterHub base:519] Setting cookie oauthenticator-state: {'httponly': True, 'secure': True, 'expires_days': 1}
[I 2021-01-21 16:55:33.539 JupyterHub log:181] 302 GET /hub/oauth_login?next= -> https://login.microsoftonline.com/e00ddcdf-1e0f-4be5-a37a-894a4731986a/oauth2/authorize?response_type=code&redirect_uri=https%3A%2F%2Fvm-dsvmv2-ihprodtest2.bdaa.prod.ams.azu.dbgcloud.io%3A8000%2Fhub%2Foauth_callback&client_id=de9b193b-6b87-4ee8-954e-de1e6e6d99d9&state=[secret] (@::ffff:10.250.6.187) 1.94ms
[E 2021-01-21 16:55:33.974 JupyterHub web:1789] Uncaught exception GET /hub/oauth_callback?code=0.AAAA39wN4A8e5UujeolKRzGYajsZm96Ha-hOlU7eHm5tmdlHAOQ.AQABAAIAAABeStGSRwwnTq2vHplZ9KL4YTkNi3qqCQtVs2wQum5strzM01iQSJTQacPyjTFmvU3OxGWYIOHTje2sbORUYEiTuLpoL262hrjry-NjGj5Ae7r_t8fFKlJELHtB5ij2HkG2eka6CIe-GBNGRLXC7aF1j99COAG9DRck5q38tcm9T8CLGr_HfseI3fzCiOzwU7kNeUSAl-gB8X8fuq6nz5jYN49CeSJBRn1_4w3EqQg9fY39B3qCUwLSmLVSWlJiWnxyXA-I-OKRc7QXEIN0Sz3ZwRoAZ6lSZEg6P_3zLdj0p_Xei_G3VieRsSu5b0JAv9pU696drT_okqWT1w3ILqix2SiClCzrYDCrakpBVdsYNGT0_-1zj-0e4QHdeQd3Cv6TQRa2JXVbmfIBAvNtRMQ-SAu_vXlnCfm3qTRlfti8xMMfvhHa3mJv-zeqlAgJSiGFDAvVhsCIc21o8cCzxNm4s7ui2TVBKfvW-HgK43opdGI6gCqtE3_C3katJR8HZxWtQD3oId3AQKCROmIHjVNwNcMkfsolouQXuL14KvsBKXjSwKJwX24jxW-hU8p8rZD4RJwKlyjhyB7WV2xAOgxXzkW2fQck-jWteM1EOrLjhUJVFbAPzoPohCKxkDdPHfqRFqNB94z1_VkSeglukajMAV2ZHwbVDC0C9imhbkJKMPbJWnBhwA81ywjJWCCYd1ogdSgJxFl3yd4fR-9mGog6zQJnldFyn3D8W6ifSvNYmCAA&state=eyJzdGF0ZV9pZCI6ICI0ODFkMDQxMzBjNzE0NTI3YjkwZGJkMDNjOTMzMmQ2ZSIsICJuZXh0X3VybCI6ICIifQ%3d%3d&session_state=47babd93-3a49-4128-9bd2-05e3b2987279 (::ffff:10.250.6.187)
    HTTPServerRequest(protocol='https', host='vm-dsvmv2-ihprodtest2.bdaa.prod.ams.azu.dbgcloud.io:8000', method='GET', uri='/hub/oauth_callback?code=0.AAAA39wN4A8e5UujeolKRzGYajsZm96Ha-hOlU7eHm5tmdlHAOQ.AQABAAIAAABeStGSRwwnTq2vHplZ9KL4YTkNi3qqCQtVs2wQum5strzM01iQSJTQacPyjTFmvU3OxGWYIOHTje2sbORUYEiTuLpoL262hrjry-NjGj5Ae7r_t8fFKlJELHtB5ij2HkG2eka6CIe-GBNGRLXC7aF1j99COAG9DRck5q38tcm9T8CLGr_HfseI3fzCiOzwU7kNeUSAl-gB8X8fuq6nz5jYN49CeSJBRn1_4w3EqQg9fY39B3qCUwLSmLVSWlJiWnxyXA-I-OKRc7QXEIN0Sz3ZwRoAZ6lSZEg6P_3zLdj0p_Xei_G3VieRsSu5b0JAv9pU696drT_okqWT1w3ILqix2SiClCzrYDCrakpBVdsYNGT0_-1zj-0e4QHdeQd3Cv6TQRa2JXVbmfIBAvNtRMQ-SAu_vXlnCfm3qTRlfti8xMMfvhHa3mJv-zeqlAgJSiGFDAvVhsCIc21o8cCzxNm4s7ui2TVBKfvW-HgK43opdGI6gCqtE3_C3katJR8HZxWtQD3oId3AQKCROmIHjVNwNcMkfsolouQXuL14KvsBKXjSwKJwX24jxW-hU8p8rZD4RJwKlyjhyB7WV2xAOgxXzkW2fQck-jWteM1EOrLjhUJVFbAPzoPohCKxkDdPHfqRFqNB94z1_VkSeglukajMAV2ZHwbVDC0C9imhbkJKMPbJWnBhwA81ywjJWCCYd1ogdSgJxFl3yd4fR-9mGog6zQJnldFyn3D8W6ifSvNYmCAA&state=eyJzdGF0ZV9pZCI6ICI0ODFkMDQxMzBjNzE0NTI3YjkwZGJkMDNjOTMzMmQ2ZSIsICJuZXh0X3VybCI6ICIifQ%3d%3d&session_state=47babd93-3a49-4128-9bd2-05e3b2987279', version='HTTP/1.1', remote_ip='::ffff:10.250.6.187')
    Traceback (most recent call last):
      File "/datadrive/anaconda/lib/python3.8/site-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/datadrive/anaconda/bin/oauthenticator-master/oauthenticator/oauth2.py", line 224, in get
        user = await self.login_user()
      File "/datadrive/anaconda/lib/python3.8/site-packages/jupyterhub/handlers/base.py", line 747, in login_user
        authenticated = await self.authenticate(data)
      File "/datadrive/anaconda/lib/python3.8/site-packages/jupyterhub/auth.py", line 459, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
      File "/datadrive/anaconda/bin/oauthenticator-master/oauthenticator/azuread.py", line 75, in authenticate
        resp = await http_client.fetch(req)
    tornado.curl_httpclient.CurlError: HTTP 599: SSL certificate problem: unable to get local issuer certificate

[D 2021-01-21 16:55:33.976 JupyterHub base:1256] No template for 500
[E 2021-01-21 16:55:33.981 JupyterHub log:173] {
      "X-Forwarded-Host": "vm-dsvmv2-ihprodtest2.bdaa.prod.ams.azu.dbgcloud.io:8000",
      "X-Forwarded-Proto": "https",
      "X-Forwarded-Port": "8000",
      "X-Forwarded-For": "::ffff:10.250.6.187",
      "Cookie": "_xsrf=[secret]; oauthenticator-state=[secret]",
      "Accept-Language": "en-US,en;q=0.9",
      "Accept-Encoding": "gzip, deflate, br",
      "Referer": "https://vm-dsvmv2-ihprodtest2.bdaa.prod.ams.azu.dbgcloud.io:8000/",
      "Sec-Fetch-Dest": "document",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Site": "cross-site",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
      "User-Agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "Connection": "close",
      "Host": "vm-dsvmv2-ihprodtest2.bdaa.prod.ams.azu.dbgcloud.io:8000"
    }
[E 2021-01-21 16:55:33.981 JupyterHub log:181] 500 GET /hub/oauth_callback?code=[secret]&state=[secret]&session_state=[secret] (@::ffff:10.250.6.187) 117.74ms
[D 2021-01-21 16:55:34.047 JupyterHub log:181] 200 GET /hub/static/css/style.min.css?v=a690a7f26f8eda24b6043d972eda379b7f3df65dcb4832fbec0d72d1585b6bdf2bd15c7f3ec4597b79a7c91798181ba23b9a0204da8165b21ff81cd85f89a933 (@::ffff:10.250.6.187) 1.07ms

however, when we disable proxy setup as below

http_proxy=""
https_proxy=""
no_proxy=""

Jupyter oauth login works fine.

We are seeing this issue only with recent versions (EDIT: 0.12.4) of Jupyterhub-oauthenticator and this worked fine before (EDIT: 0.11.1).

Your personal set up

• OS: REDHAT 7.7
• Python: 3.8.3

jupyter --version

jupyter --version
jupyter core     : 4.7.0
jupyter-notebook : 6.2.0
qtconsole        : not installed
ipython          : 7.12.0
ipykernel        : 5.4.2
jupyter client   : 6.1.11
jupyter lab      : 2.2.9
nbconvert        : 6.0.7
ipywidgets       : not installed
nbformat         : 5.1.2
traitlets        : 5.0.5

jupyterhub_config.py

# jupyterhub_config.py

c.Authenticator.admin_users = {'xxx'}
c.Authenticator.whitelist = {'xxx'}
from oauthenticator.azuread import AzureAdOAuthenticator
c.JupyterHub.authenticator_class = AzureAdOAuthenticator
c.AzureAdOAuthenticator.oauth_callback_url = 'https://xxx/hub/oauth_callback'
c.AzureAdOAuthenticator.client_id = 'xxx'
c.AzureAdOAuthenticator.tenant_id = 'xxx'
c.AzureAdOAuthenticator.client_secret = 'xxx'
c.AzureAdOAuthenticator.tls_verify='0'
c.JupyterHub.log_level = 'DEBUG'
c.ConfigurableHTTPProxy.api_url = 'xxx'
c.JupyterHub.ssl_cert = 'xxx'
c.JupyterHub.ssl_key = 'xxx'
c.Spawner.default_url = '/lab'
c.Spawner.notebook_dir = '~/notebooks'

[mvuddaraju]

Hello @manics ,

Any updates on the issue here?.

Regards, Madhuri

[consideRatio]

This is probably related https://github.com/jupyterhub/oauthenticator/issues/217.