ImageRef is mutated with the wrong digest
See original GitHub issueDescribe the bug
This issue may be precipitated by https://github.com/sse-secure-systems/connaisseur/issues/582. Our CI platform routinely deploys about 40 pods into a fresh namespace to test code changes. We noticed that after deploying connaisseur, some of the time, the pods in these namespaces fail with an error like this:
Warning Failed 12s kubelet Failed to pull image "docker.io/gergorg/rtdb@sha256:8f76cb854b018213597d910fb4a1ad7d5d5ed775fbe328594b9d6347a48c38f9": rpc error: code = NotFound
desc = failed to pull and unpack image "docker.io/myorg
/rtdb@sha256:8f76cb854b018213597d910fb4a1ad7d5d5ed775fbe328594b9d6347a48c38f9": failed to resolve reference
"docker.io/myorg/rtdb@sha256:8f76cb854b018213597d910fb4a1ad7d5d5ed775fbe328594b9d6347a48c38f9": docker.io
/myorg/rtdb@sha256:8f76cb854b018213597d910fb4a1ad7d5d5ed775fbe328594b9d6347a48c38f9: not found
The digest ref is the result of a deployment being mutated by connaisseur, having the image tag changed into the digest format.
However… the actual digest applied belongs to a different image in a different repository, which was deployed by CI at the same time into the namespace:

It would seem that connaisseur “mixed up” the digests of the various deployments/pods which were deployed at the same time (potentially due to an error parsing the CRs), and substituted the digest for one image, with another.
Expected behavior
Connaisseur only mutates imageRefs matching the repository and image which was originally intercepted
Optional: To reproduce
This is hard, since the issue seems to be intermittent. It seems to co-inside with the errors generated by https://github.com/sse-secure-systems/connaisseur/issues/582. To reproduce, try deploying about 20 different images simultaneously, mixed in with some CRDs which include an image ref.
Optional: Versions (please complete the following information as relevant):
- Connaisseur: 2.5.1
Issue Analytics
- State:
- Created 2 years ago
- Comments:13 (5 by maintainers)
Thanks heaps, @xopham,
-rc3
seems to work as expected, under high load conditions 😃Hey @xopham, I’m assuming the only difference is the image (
xoph/connaisseur:v2.5.2-rc
)? If so, I’ll substitute it into my existing connaisseur deployment and test 😃