Feature Request: Default pod affinity

Dask-kubernetes is currently configured to include two default pod/taint tolerations (see #109, #110, #145). This feature allows dask-worker pods to be the sole inhabitants of a node/node-pool, provided the cluster is configured with the corresponding taints. Another scheduling pattern would be to attract dask-worker pods to a specific set of nodes. While this has been done successfully with nodeSelectors in the past, it is not a particularly user-friendly option.

I’m wondering if we could come up with a universal dask-worker label and then have dask-kubernetes set a preferredDuringSchedulingIgnoredDuringExecution affinity on each pod. Has anyone tried this? Are there specific downsides to doing this? My understanding is that the nodeSelector approach will be eventually deprecated in favor of affinity rules.

I think something like this would work (haven’t tested):

spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: k8s.dask.org/node-purpose
            operator: In
            values:
            - worker

The main benefits here are:

1. This doesn’t require users/admins to set anything, existing applications without labels would continue to function as normal
2. It would allow admins to direct traffic to specific nodes without asking the user to configure their pod template

Thoughts from @yuvipanda, @jacobtomlinson, @betatim?

[jhamman]

Okay, I see where you are headed. I think we should start with a default prefer affinity. Then follow up with making the prefer vs require option configurable at the dask kubernetes config level.

[jhamman]

closed by #147