A username with @ --> Invalid k8s label

A user attempting to create a cluster (JupyterHub authentication) with @ in their username, will experience an error. This is because at some point an invalid Kubernetes resource is being created due to this, the invalid part is a label that has a value with the username, and a Kubernetes label values isn’t allowed to contain @.

How I reproduce the issue

from dask_gateway import Gateway
# jupyterhub auth is configured in /etc/dask/gateway.yaml
# my username is [e-mail]
gateway = Gateway(address="http://api-dask-gateway.dask-gateway:8000")
cluster = gateway.new_cluster()

Logs from deploy/api

[I 2020-04-09 11:03:16.794 DaskGateway] Creating cluster dask-gateway.c8451c67dc094eb1ad9918b8260c527a for user [e-mail]
[E 2020-04-09 11:03:16.813 aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/opt/conda/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/opt/conda/lib/python3.7/site-packages/dask_gateway_server/routes.py", line 13, in inner
    return await handler(request)
  File "/opt/conda/lib/python3.7/site-packages/dask_gateway_server/routes.py", line 47, in inner
    return await auth.authenticate_and_handle(request, handler)
  File "/opt/conda/lib/python3.7/site-packages/dask_gateway_server/auth.py", line 102, in authenticate_and_handle
    response = await handler(request)
  File "/opt/conda/lib/python3.7/site-packages/dask_gateway_server/routes.py", line 124, in create_cluster
    cluster_name = await backend.start_cluster(user, cluster_options)
  File "/opt/conda/lib/python3.7/site-packages/dask_gateway_server/backends/kubernetes/backend.py", line 343, in start_cluster
    "gateway.dask.org", self.crd_version, config.namespace, "daskclusters", obj
  File "/opt/conda/lib/python3.7/site-packages/kubernetes_asyncio/client/api_client.py", line 166, in __call_api
    _request_timeout=_request_timeout)
  File "/opt/conda/lib/python3.7/site-packages/kubernetes_asyncio/client/rest.py", line 230, in POST
    body=body))
  File "/opt/conda/lib/python3.7/site-packages/kubernetes_asyncio/client/rest.py", line 181, in request
    raise ApiException(http_resp=r)
kubernetes_asyncio.client.rest.ApiException: (422)
Reason: Unprocessable Entity
HTTP response headers: <CIMultiDictProxy('Audit-Id': 'd2c5666f-def5-4989-b221-0f40fce9137c', 'Content-Type': 'application/json', 'Date': 'Thu, 09 Apr 2020 11:03:16 GMT', 'Content-Length': '1018')>
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"DaskCluster.gateway.dask.org \"c8451c67dc094eb1ad9918b8260c527a\" is invalid: metadata.labels: Invalid value: \"[e-mail]\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')","reason":"Invalid","details":{"name":"c8451c67dc094eb1ad9918b8260c527a","group":"gateway.dask.org","kind":"DaskCluster","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"[e-mail]\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')","field":"metadata.labels"}]},"code":422}

[jcrist]

I originally used a label because you can use label selectors to filter objects by users (you can’t select by annotation).

Annotations don’t have the same restrictions as limits, perhaps we add an annotation with the username?

On second thought, I think I like not setting this annotation by default, and relying on the user to manage this themselves in the configuration (using e.g. extra_worker_pod_annotations). Internally we track the username as part of the CRD object, no need to duplicate it as an annotation if we don’t need to.

[gforsyth]

For users of kubecost, my understanding (which is very limited on this front) is that it tracks labels and annotations and sticks them into prometheus for tracking usage, etc.
kubecost doesn’t appear to be hugely extensible (this is where my limited knowledge is coming in to play) so for organizations that need to track usage on a per-user basis having the username as a label is helpful.

That said, if it breaks for certain username patterns, then leaving it as a label doesn’t make sense.

I added it back in using extra_worker_pod_labels via a cluster option which works fine on our end – would example configurations for how to set these things up be helpful in the docs?