question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Incorrectly parsed image references if they contain both a tag and digest

See original GitHub issue

Describe the bug

I use image references in my manifests that contain both a tag and a sha256 digest, e.g.: example.com/library/image:v1.2.3@sha256:abcd.... This causes the admission webhook to fail with:

Error: INSTALLATION FAILED: admission webhook "connaisseur-svc.connaisseur.svc" denied the request: Unexpected Cosign exception for image "example.com/library/image:v1.2.3@sha256:1d076650cbf2f364235e49203a98015d35443d72e74f462085ea3bcedd25cb4c:v2.15.5": Error: parsing reference: could not parse reference: example.com/library/image:v1.2.3@sha256:1d076650cbf2f364235e49203a98015d35443d72e74f462085ea3bcedd25cb4c:v2.15.5
main.go:62: error during command execution: parsing reference: could not parse reference: example.com/library/image:v1.2.3@sha256:1d076650cbf2f364235e49203a98015d35443d72e74f462085ea3bcedd25cb4c:v2.15.5

So it appears that these images are incorrectly parsed. I believe this is caused in https://github.com/sse-secure-systems/connaisseur/blob/v2.6.2/connaisseur/image.py#L30-L65 as adding the following new test data to https://github.com/sse-secure-systems/connaisseur/blob/v2.6.2/tests/test_image.py#L115 causes the tests to fail:

        (
            "docker.io/library/image:v1.2.3@sha256:f8816ada742348e1adfcec5c2a180b675bf6e4a294e0feb68bd70179451e1242",
            "image",
            "v1.2.3",
            "f8816ada742348e1adfcec5c2a180b675bf6e4a294e0feb68bd70179451e1242",
            "library",
            "docker.io",
            fix.no_exc(),
        ),

https://github.com/sse-secure-systems/connaisseur/blob/v2.6.2/connaisseur/image.py#L84 has to be adapted to either return the full tag+digest or just the digest as it is passed to cosign verify later on (https://github.com/sse-secure-systems/connaisseur/blob/v2.6.2/connaisseur/validators/cosign/cosign_validator.py#L304).

Cosign itself does support both tags+digest when verifying, but drops the tag if both a tag+digest are specified e.g.

$ COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/chgl/fhir-server-exporter:v2.0.2@sha256:0d42bd5680994b2f3e376ef1159ec89ede6b1d59892fcd0f8079672304f99f7f

Verification for ghcr.io/chgl/fhir-server-exporter@sha256:0d42bd5680994b2f3e376ef1159ec89ede6b1d59892fcd0f8079672304f99f7f --
...

Expected behavior

It should be possible to verify image references with both a tag and a digest specified. If both are present, from an implementation point of view it’s probably easiest to drop the tag and just use the digest when passing the image reference.

Optional: To reproduce

  1. Install connaisseur
  2. try deploying a manifest containing an image reference with both a tag and digest
  3. fails with the above error message as the image being passed to cosign is incorrectly constructed

Optional: Versions (please complete the following information as relevant):

  • OS: <OS and version>
  • Kubernetes Cluster: <cluster type and version>
  • Notary Server: <notary and version>
  • Container registry: <container registry and version>
  • Connaisseur: <connaisseur version>
  • Other: <other relevant tools and versions>

Optional: Additional context

As a tiny side note, in https://github.com/sse-secure-systems/connaisseur/blob/v2.6.2/connaisseur/image.py#L84 hard-coding the digest to sha256 could possibly be an issue some time in the future as technically various types of digests are possible, see https://github.com/opencontainers/image-spec/blob/main/descriptor.md#digests - although admittedly I’ve never seen anything but sha256 in the wild.

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:8 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
richgerrardcommented, Oct 18, 2022

Please consider the proposed fix at your earliest convenience, @xopham

For clarity, from the OCI spec:

- Digest: a unique identifier created from a cryptographic hash of a Blob's content. Digests are defined under the OCI Image Spec [apdx-3](https://github.com/sse-secure-systems/connaisseur/issues/749#appendix)
- Tag: a custom, human-readable manifest identifier

I’m all for immutability, but I’m a human being first and foremost. When I’m operating my enterprise clusters, my observability tools suite only display a tangled mess of digest references following the mutating webhook implementation.

My goal is to know what is deployed in my cluster without wasting time. My human brain knows that tags are mutable. My human brain knows that tags are not semvers. My human brain prefers a shorthand for identifying a manifest. My human brain is why we wish to include tags AND digests in manifest files together.

You can already sort out the validation based on the immutable part, and it’s perfectly clear that this is a bug that requires an immediate fix. I’ll argue later if your mutating webhook also fails to preserve my tag value, but for now: please consider releasing a fix for this bug at your earliest convenience.

It’s for the humans. Thank you!

1reaction
xophamcommented, Aug 16, 2022

An interesting situation and surprising that it exists 😅 Without having delved into the details, it seems to me that we have no standard to follow here and one is generally free to do as one likes. That makes me a bit hesitant to be creative, as it is hard to know what people would expect to happen. If we were to support this, there is essentially 2 questions to answer if tag+digest are provided:

  1. how to read it if both are provided: Drop the tag? Verify tag to digest? Drop the digest? Just pass on to the validator (notaryv1 or cosign)?
  2. how do we interpret and apply the result: Deny or overwrite if tag and digest do not match? As there is a digest, do we only give a yes or no answer whether the digest checks out?

I’d presume that the best way forward may be to pass the whole construct to the validator and go for only validating but not modifying the request as there is a digest and the user intended to have that construct validated and the user would have to take care that it can actually be interpreted by the container runtime which I’d rather not want to take care of.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Docker references with both a tag and digest are currently not ...
1, interpret this as “ask for the image with the specified tag, then fail if the manifest does not match the specified digest”....
Read more >
batch-get-image — AWS CLI 1.27.32 Command Reference
Gets detailed information for an image. Images are specified with either an imageTag or imageDigest . When an image is pulled, the BatchGetImage...
Read more >
Support containers with name:tag@digest
Issue. s2i builds referring to containers by both tags and SGA256 digest fail. For example: Raw.
Read more >
containers-signature(5) — golang-github-containers-image ...
As distributed, the container signature is a blob which contains a ... when pulling the :latest tag or when referencing an image by...
Read more >
What is Text Analysis? A Beginner's Guide - MonkeyLearn
If you receive huge amounts of unstructured data in the form of text (emails, ... As you can see in the images above,...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found