Kubernetes: Allow to specify NodeSelector and/or Affinity for Runners

Yesterday I’ve installed Predator for the first time and launched my first test. Good job on turning Artillery into distributed load test that runs as kubernetes Jobs! ❤️

I wanted to test the HPA for some of my application to make sure that HPA was setup right and see how much pounding they could take.

In order to make sure the tests were not interferring with the application being tested, I’ve created a new GKE node pool with a specific label purpose:loadtesting and I’ve configured my app to never be scheduled on one of those nodes (using affinity).

Here is how I created the node pool in GKE:

    gcloud container node-pools create ${POOL_NAME} \
           --cluster ${CLUSTER} \
           --enable-autoscaling \
           --machine-type n1-standard-4 \
           --max-nodes 12 \
           --min-nodes 3 \
           --node-version 1.15.11-gke.5 \
           --num-nodes 4 \
           --scopes gke-default,logging-write,monitoring \
           --zone ${ZONE} \
           --node-labels=purpose=loadtesting

Then I wanted the tests (runner) to run ONLY on nodes which had the label purpose:loadtesting.

So that’s my suggestion, being able to force (using nodeSelector for example) the runner pods on specific nodes (or prevent them to run on some nodes using affinity).

I was able to install the predator helm chart and specify that I wanted that to run in my loadtesting nodes by using:

helm install predator zooz/predator --set nodeSelector.purpose=loadtesting

and that worked just fine. But that is not having any impact on the runner.

So what I ended up doing is to clone your repo, and I made a small change to src/jobs/models/kubernetes/jobTemplate.js and now the runners runs on the nodes I want.

The diff looks like:

--- a/src/jobs/models/kubernetes/jobTemplate.js
+++ b/src/jobs/models/kubernetes/jobTemplate.js
@@ -30,6 +30,9 @@ module.exports.createJobRequest = (jobName, runId, parallelism, environmentVaria
                             'env': Object.keys(environmentVariables).map(environmentVariable => ({ name: environmentVariable, value: environmentVariables[environmentVariable] }))
                         }
                     ],
+                    'nodeSelector': {
+                        'purpose': 'loadtesting'
+                    },
                     'restartPolicy': 'Never'
                 }
             },

But this only works for my case. It would be nice if the helm chart values had a section called runners where I could specify things like nodeselector or affinity or tolerations which would be applied to the runner Jobs.

The alternative would be to change the UI to ask for these settings, but… since that only apply to kubernetes, I thought it made more sense to have this as part of the chart configurable values…

Let m know if you need any more details.

[enudler]

Hi @NivLipetz @roychri I would like to start working on that one.

Proposal: Add a configmap to predator helm chart where you can define the runner params: for example:

apiVersion: v1 kind: ConfigMap metadata: name: predator-runner-configmap data: template: |- { "spec": { "template": { "metadata": { "annotations": { "traffic.sidecar.istio.io/excludeOutboundPorts": "8060" } } } } }

Predator will merge the given template of the runner with its own current ‘hardcoded’ job template. this will give good flexibility for future uses and adjustments.

What do you think?

[enudler]

@roychri although the issue is closed I realized that I put a wrong example so example updated 😃