Add MINIMAL example of how to deploy on Google Cloud

I personally am having a bit of trouble deploying my dask workload on GKE. I doubt I’m the only one, as the current “quickstart” guide makes a lot of assumptions and has a lot of extra stuff (helm, jupyterhub) that confuses the core ideas for those not familiar with gce deployments or kubernetes.

I suggest adding a minimal deployment tutorial that includes:

• Setting up a gke cluster (already covered by zero-to-jupyterhub)

• What about local ssds? Does each pod require a local ssd if the underlying datasets exceed worker memory. Or, does the dask-worker-space reside on the “master” node? Do the pods have access the master nodes disk?

• Creating/linking a docker image for the worker

• Can I build the image on the “master” node and the workers will be able to find it?

• Writing a yaml file for the gke cluster and docker image (not covered in any detail, from what I can find).

Here is the toy script I’m working with to try to get running on GKE:

from dask.distributed import Client,get_worker
from dask_kubernetes import KubeCluster

def write_file(idx):
    fName = "file_{:04d}.txt".format(idx);
    content = fName + " " + str(get_worker().id) + "\n";
    with open(fName,'w+') as f:
        f.write(content);
    return (fName,content);

if __name__=="__main__":
    cluster = KubeCluster();
    client = Client(cluster);
    futs = []
    for ii in range(0,10):
        futs.append( client.submit(write_file,ii) );

    print(futs,'\n\n')
    out = client.gather(futs);
    print(out)

Dockerfile:

FROM python:3.6
RUN mkdir /app
WORKDIR /app
ADD . /app/
RUN pip install -r requirements.txt

Any help would be appreciated!

[jacobtomlinson]

You need to create a service account for that pod with the correct RBAC perms. We have this config over in the Pangeo project which adds dask-kubernetes support to Zero to JupyterHub.

[bgoodman44]

Thank you Jacob! It works and it is GLORIOUS! I still have a few things to hash out, but than I’ll share my “how-to”.