[bug] `JUPYTERHUB_API_TOKEN` incorrectly set for prefect pods

Describe the bug

A clear and concise description of what the problem is.

After enabling the prefect-agent and registering a few flows with it, and after getting past the issues described in #707 (by creating a much smaller image), we are still unable to run a flow from end-to-end. The error we are consistently seeing when trying to run a flow is the following: dask_gateway.client.GatewayServerError: 500 Internal Server Error

Given all the debugging and testing I’ve done to this point, I have strong reason to believe that the error arises from how the JUPYTERHUB_API_TOKEN is set for all of the pods besides jupyterlab instances.

To highlight the potential bug more clearly see the print out below. Note, this is while attached to the prefect-job pod:

### See JUPYTERHUB_API_TOKEN ###
(prefect) root@prefect-job-7e35df5b-pkdmv:/app# env | grep JUPYTERHUB_API
JUPYTERHUB_API_TOKEN=<SAME API TOKEN AS PREFECT-AGENT/DASKGATEWAY-GATEWAY PODS> 
(prefect) root@prefect-job-7e35df5b-pkdmv:/app# python
Python 3.8.10 | packaged by conda-forge | (default, May 11 2021, 07:01:05) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dask_gateway import Gateway
>>> g = Gateway()
>>> g.cluster_options()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/envs/prefect/lib/python3.8/site-packages/dask_gateway/client.py", line 518, in cluster_options
    return self.sync(
  File "/opt/conda/envs/prefect/lib/python3.8/site-packages/dask_gateway/client.py", line 343, in sync
    return future.result()
  File "/opt/conda/envs/prefect/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/opt/conda/envs/prefect/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/opt/conda/envs/prefect/lib/python3.8/site-packages/dask_gateway/client.py", line 497, in _cluster_options
    resp = await self._request("GET", url)
  File "/opt/conda/envs/prefect/lib/python3.8/site-packages/dask_gateway/client.py", line 417, in _request
    raise GatewayServerError(msg)
dask_gateway.client.GatewayServerError: 500 Internal Server Error

Server got itself in trouble
>>> exit()
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f1d11f97d30>

### JUPYTERHUB_API_TOKEN changed ###
(prefect) root@prefect-job-7e35df5b-pkdmv:/app# export JUPYTERHUB_API_TOKEN=<COPIED FROM USER JUPYTERLAB POD> 
(prefect) root@prefect-job-7e35df5b-pkdmv:/app# python
Python 3.8.10 | packaged by conda-forge | (default, May 11 2021, 07:01:05) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dask_gateway import Gateway
>>> g = Gateway()
>>> g.cluster_options()
<dask_gateway.options.Options object at 0x7f61f3cd78b0>

I’ve tried digging into how the JUPYTERHUB_API_TOKEN is set for the user jupyterlab instance vs prefect-agent but I don’t have any definitive answers. I think each of the pods has it’s JUPYTERHUB_API_TOKEN set accordingly:

• daskgateway-gateway - here
  • It is also hidden in the daskgateway-gateway pod yaml but with a format that looks off:

  env:
  - name: JUPYTERHUB_API_TOKEN 
    valueFrom:
      secretKeyRef:
        key: jupyterhub_api_token
        name: qhub-daskgateway-gateway
        optional: false 
  

• prefect - here
  • It is also hidden in the prefect-agent pod yaml but exposed on in the prefect-job pod
  • The prefect-agent pod is running on in the general node group whereas the prefect-job pod is running in the user_worker node group
• jupyterhub - here
  • This seems to conform to the dask gateway documentation and seemds to be relabeled as JUPYTERHUB_API_TOKEN

Each time I spin up a jupyterlab instance I get a new JUPYTERHUB_API_TOKEN however, the same cannot be said for the prefect-agent, prefect-job or qhub-daskgateway-gateway pods. Even after killing these pods manually, they appear to retain the same JUPYTERHUB_API_TOKEN and what’s more, this token appear to be the base64 decoded jupyterHubToken secret.

At this point, I’ve run up against my limited Terraform knowledge. Do you happen to have an insights or ideas on how to get the JUPYTERHUB_API_TOKEN properly set on the prefect-agent and prefect-job?

What I have tried thus far?

• Manually exporting the token while attached to those pods (like I did in the above example), however when I disconnect from the pod, the JUPYTERHUB_API_TOKEN reverts back.
• Generating a token from the JupyterHub Control Panel and hard coding it into the flow. This quick and dirty approach allows us to spin up a dask cluster.

Your environment

Describe the environment in which you are experiencing the bug.

Include your conda version (use conda --version), k8s and any other relevant details.

[aktech]

It’s fixed in dask gateway, we’re waiting for the next release of dask-gateway to close this.

[github-actions[bot]]

This issue was closed because it has been stalled for 7 days with no activity.