Support random port assignment when c.KubeSpawner.port = 0

According to the document, KubeSpawner will use randomly allocated port. I deployed zero-to-jupyterhub-k8s (hostNetwork:true set for spark) and got error when login some users: [Warning] 0/3 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 2 Insufficient memory, 2 node(s) didn't have free ports for the requested pod ports.

That’s because KubeSpawner always use port 8888 instead of random port. https://github.com/jupyterhub/kubespawner/blob/master/kubespawner/spawner.py#L145

https://jupyterhub-kubespawner.readthedocs.io/en/latest/spawner.html

config c.KubeSpawner.port = Int(0) The port for single-user servers to listen on.

Defaults to 0, which uses a randomly allocated port number each time.

If set to a non-zero value, all Spawners will use the same port, which only makes sense if each server is on a different address, e.g. in containers.

New in version 0.7.

[frouzbeh]

@ramkrishnan8994 My Kubernetes and Hadoop cluster are physically on the same computer cluster. I thought to connect to spark from client side I needed hostNetwork but I don’t. I just needed to set the spark.driver.host of SparkConf to the ip address of the container.

[cmd-ntrf]

As @rkdarst mentionned, we encountered a similar issue with batchspawner.

The solution we opted for was to write a API Handler that is installed on the Hub side. The handler waits to receive the port number from the singleuser and modify the spawner port value. The spawner is identified based on the user auth, but I have recently submitted a patch to use the API token instead to support named servers.

To send the port, I have written a small wrapper script that selects a port, configures the singleuser to use it, send it through http to the Hub at the API handler address, then starts the notebook just like singleuser would.

There is a problem though. JupyterHub does not provide mechanism to automatically registers API handlers from third-parties. Currently, the API handler is registered when the batchspawner module is imported, but for some cases like when using wrapspawner, the module is imported after JupyterHub is initialized and the batchspawner API handler is not registered properly. As a solution to that problem, we currently instructing user to import batchspawner in jupyterhub_config.py, which is not ideal, but it works.

Ideally, the API handler I have written for batchspawner would be integrated directly in JupyterHub to configure the port number. Another option would be to implement a mechanism similar to the one in Jupyter that allow the installation and activation of server side plugin / handlers. I am willing to help with either solution or anything related.