Access to docker/runtime options (PID limits)

I’ve been triaging some stability issues with our kubernetes cluster (short version, don’t use centos7 and/or recompile containerd , or you’ll eventually run out of kernel memory and the host will start locking up). After experimenting, I found that a fork bomb triggers the issue quickly. While the kernel memory leak can be fixed/mitigated, I’m now worried about DoS via PID exhaustion, accidental or not.

Docker seems to have PID limits via the run flag --pids-limit. Is it possible to access other docker commandline options like this through the spawner API?

I appreciate the help! I wouldn’t be surprised if the low level argument control is firewalled from the API.

Perhaps pid limits can be enforced inside the container via ulimit, but I’m not sure since that seems like something that would involve the kernel.

[vilhelmen]

Thanks for the reminder! I got absolutely nowhere trying to fight kubespawner to accept my config. I didn’t really want to mod the spawner to try and make it accept it. I’m very bad at k8s.

But, more importantly, I have a BIG WARNING for anyone that tries to go the ulimit route: You will super duper mess up jupyter sessions and probably need to set it to like 500+

I tried setting it to 200 and student sessions would eventually grind to a halt. The process limit counts threads and the jupyter worker threads add up incredibly quickly. Jupyter doesn’t fail gracefully when it can’t spawn threads, buttons just stop doing things.

[consideRatio]

Kubernetes Kubelet, which runs on each k8s node, can be started with --pod-max-pids it seems, but that is not something that KubeSpawner can configure as part of registering a Pod with the k8s api-server. I’ll go ahead and close this issue as unresolvable as part of this project.