Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Question: Tasks not getting cached

See original GitHub issue

👋 I am new to Kubeflow, I used kfctl to install kbf. Pipelines work fine, but caching is not working. Tasks never get skipped. There is nothing in the logs either:

kubectl logs -n kubeflow --selector app=cache-server
2020/10/28 17:13:34 Initing client manager....
2020/10/28 17:13:34 Database created
2020/10/28 17:13:34 execution_caches

I made sure that mutatingwebhookconfiguration exists and that is setup as mentioned in https://www.kubeflow.org/docs/pipelines/caching/

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:22 (13 by maintainers)

Top GitHub Comments

1reaction

cah4444commented, Dec 1, 2020

I figured out the problem. When i deleted the kubeflow pipelines deployment to recreate is from scratch following This, it left behind the mutatingwebhookconfiguration for the namespace (I wasn’t aware that it was a cluster-wide resource). Manually deleting that before reinstalling fixed the cache server issue.

I imagine it had been left from when i had done the kfp uninstall/reinstall before, and had mismatch between TLS certs. Not certain of this, however.

Also, i saw no events or errors in status messages on either of the two resources.

Thank you for all of your help!

1reaction

cah4444commented, Nov 23, 2020