Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Reflection is to slow. Pod goes into ImagePullBackOff

See original GitHub issue

I am using Gitops and on pull requests of my app I create a new environment to test the PR. I will create a namespace and the required pods. But until the kubernetes-reflector replicated the imagepullsecret, the pods already went into ImagePullBackOff state.

kubernetes-reflector logs show:

2022-05-04 12:38:13.495 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Session closed. Duration: 00:35:19.1644564. Faulted: False.
2022-05-04 12:38:13.504 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Requesting V1Secret resources
2022-05-04 12:38:13.768 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected my-ns/my-secret where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 9.
2022-05-04 13:12:02.234 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Session closed. Duration: 00:57:19.8038623. Faulted: False.
2022-05-04 13:12:02.235 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Requesting V1ConfigMap resources
2022-05-04 13:15:50.371 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Session closed. Duration: 00:38:42.5296269. Faulted: False.
2022-05-04 13:15:50.372 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Requesting V1Namespace resources
2022-05-04 13:37:29.214 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Session closed. Duration: 00:59:15.7097240. Faulted: False.
2022-05-04 13:37:29.214 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Requesting V1Secret resources
2022-05-04 13:37:29.554 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected my-ns/my-secret where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 9.
2022-05-04 13:44:03.810 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Session closed. Duration: 00:32:01.5747240. Faulted: False.
2022-05-04 13:44:03.810 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Requesting V1ConfigMap resources
2022-05-04 14:03:22.530 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Session closed. Duration: 00:47:31.9659988. Faulted: False.
2022-05-04 14:03:22.530 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Requesting V1Namespace resources
2022-05-04 14:14:12.683 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Session closed. Duration: 00:30:08.6807986. Faulted: False.
2022-05-04 14:14:12.683 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Requesting V1ConfigMap resources
2022-05-04 14:30:02.454 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Session closed. Duration: 00:52:33.0470167. Faulted: False.
2022-05-04 14:30:02.454 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Requesting V1Secret resources
2022-05-04 14:30:02.562 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected my-ns/my-secret where permitted. Created 1 - Updated 0 - Deleted 0 - Validated 9.
2022-05-04 14:30:02.600 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created my-new-ns/my-secret as a reflection of my-ns/my-secret
2022-05-04 14:39:40.142 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Session closed. Duration: 00:36:17.6118289. Faulted: False.
2022-05-04 14:39:40.142 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Requesting V1Namespace resources

As I see it, the reflector tried to reflect secrets at 12:38, 13:37 and 14:30. So in the worst case, I’d have to wait for almost an hour until the secret get reflected.

Is this how the reflector works, or have I done something wrong? Can the reflector maybe get notified when a new namespace has been created and then “instantly” reflect secrets? Or alternatively can it check for new namespaces every 30 seconds or so?