question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

ImageRef is mutated with the wrong digest

See original GitHub issue

Describe the bug

This issue may be precipitated by https://github.com/sse-secure-systems/connaisseur/issues/582. Our CI platform routinely deploys about 40 pods into a fresh namespace to test code changes. We noticed that after deploying connaisseur, some of the time, the pods in these namespaces fail with an error like this:

  Warning  Failed       12s                kubelet            Failed to pull image "docker.io/gergorg/rtdb@sha256:8f76cb854b018213597d910fb4a1ad7d5d5ed775fbe328594b9d6347a48c38f9": rpc error: code = NotFound
 desc = failed to pull and unpack image "docker.io/myorg
/rtdb@sha256:8f76cb854b018213597d910fb4a1ad7d5d5ed775fbe328594b9d6347a48c38f9": failed to resolve reference 
"docker.io/myorg/rtdb@sha256:8f76cb854b018213597d910fb4a1ad7d5d5ed775fbe328594b9d6347a48c38f9": docker.io
/myorg/rtdb@sha256:8f76cb854b018213597d910fb4a1ad7d5d5ed775fbe328594b9d6347a48c38f9: not found

The digest ref is the result of a deployment being mutated by connaisseur, having the image tag changed into the digest format.

However… the actual digest applied belongs to a different image in a different repository, which was deployed by CI at the same time into the namespace:

Harbor 2022-03-17 16-26-56

It would seem that connaisseur “mixed up” the digests of the various deployments/pods which were deployed at the same time (potentially due to an error parsing the CRs), and substituted the digest for one image, with another.

Expected behavior

Connaisseur only mutates imageRefs matching the repository and image which was originally intercepted

Optional: To reproduce

This is hard, since the issue seems to be intermittent. It seems to co-inside with the errors generated by https://github.com/sse-secure-systems/connaisseur/issues/582. To reproduce, try deploying about 20 different images simultaneously, mixed in with some CRDs which include an image ref.

Optional: Versions (please complete the following information as relevant):

  • Connaisseur: 2.5.1

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:13 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
funkypenguincommented, Mar 24, 2022

Thanks heaps, @xopham, -rc3 seems to work as expected, under high load conditions 😃

1reaction
funkypenguincommented, Mar 22, 2022

Hey @xopham, I’m assuming the only difference is the image (xoph/connaisseur:v2.5.2-rc)? If so, I’ll substitute it into my existing connaisseur deployment and test 😃

Read more comments on GitHub >

github_iconTop Results From Across the Web

llbsolver: support pinning sources (same as the "Dockerfile.pin ...
You probably need to keep mapping of original digest to new digest after ... image ref from a registry to a local oci-layout://...
Read more >
Mutate for Digest - Bioinformatics.org
Mutate for Digest accepts a DNA sequence as input and searches for regions that can easily be mutated to create a restriction site...
Read more >
A toolkit for rapid gene mapping in the nematode ... - NCBI
The utility of polymorphisms in genetic studies was demonstrated by successful mapping of 12 mutations, including 5 that were localized to ...
Read more >
Genome-wide analysis of DNA copy-number changes using ...
In brief, we PCR-amplified IMAGE (ref. 25) human cDNAs (ESTs) in 96-well format from DNA minipreps (Qiagen) using modified M13 universal primers.
Read more >
Marshall, OW - ERIC
t onpigated a verb Their control over the language of Prodigy was fatally pedantic allowing no mutations no moditit ations no.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found