Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

K8SRunLauncher should allow passing arbitrary config to jobs/pods/containers

See original GitHub issue

Currently, K8SRunLauncher does not allow passing in custom config when creating Job/Pod/Container Kubernetes Object. This can cause problems when users want to pass in custom annotations[1], labels or any other config to their K8S Objects.

Ideas are being considered to implement this but needs prioritization: https://dagster.slack.com/archives/C014N0PK37E/p1597248672084000?thread_ts=1597019607.076100&cid=C014N0PK37E

[1]: sample annotation: cluster-autoscaler.kubernetes.io/safe-to-evict": "false" https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node This annotation can be used to stop Cluster-Auto scaler from evicting running dagster-pipeline pods.

Issue Analytics

State:
Created 3 years ago
Reactions:16
Comments:6 (2 by maintainers)

Top GitHub Comments

2reactions

ikendoitcommented, Aug 13, 2020

Additional use case from szalai1 on Slack: https://dagster.slack.com/archives/C014N0PK37E/p1597327142088800 Node-affinity to target the pipeline-runs on the nodes with GPU power.

0reactions

helloworldcommented, Aug 25, 2020

Awesome! Glad it worked.

@szalai1 I just wanted to confirm that you were successfully able to specify node-affinities. If so, we can close this issue.