[question] Should profile [env] vars affect header only package hashes?

Hi, while packaging a header only library (marked with header_only()), I noticed I got a different resulting package hash for different profiles I used. It turned out that the metadata bundled with the package (conaninfo.txt) contained the [env] fields of each specific profile, which in my case were the CC and CXX variables. Since I use the package_revision_mode for package_id resolution this resulted in my parallel CI job branches (one for each profile) stepping over each other since each one used their own package revision of this header lib for its downstream dependency package_id resolution, yet when they all upload their packages only the last revision of the header only lib remains usable (since they all have the same package_id). This results in the libs downstream dependency packages built with any other profile except the last one to upload needing an immediate rebuild.

I tried to picture the described scenario with the following diagram:

On this picture, the package of the header-only lib1 with with the revision zzz is uploaded last and overwrites the lib1 package with revision yyy since they share a package_id. But, lib2 on the left was already built using lib1 with package revision yyy, so the lib1 overwrite would render that version of lib2 outdated as well since now it requires a build with lib1 package revision zzz.

Because of the described issues I was wondering if it’s necessary to include the profile [env] fields into the metadata of a package marked with header_only()?

Conan version: 1.33.1

• I’ve read the CONTRIBUTING guide.

[solvingj]

@aleksa2808 Many people struggle with this (in all languages in fact). It’s a fundamental challenge for any workflow scenarios where dependencies are obtained via package manager, and people need to test changes to multiple packages at a time (including other programming languages).

There are currently two ways to address this with Conan:

1. For each commit/pull request, use the Artifactory API to provision a temporary Conan repository. Have the future steps use that as the upload location, and primary download location, falling back to the “production” repository for other packages. At the end of successful pipelines, you can promote the binaries from the “temporary repo”, to the “production repo”. It seems a bit “heavy-handed”, but it does achieve the goal. Furthermore, it’s actually the strategy that is used for Conan Center PRs, so we can say that it works fine with Artifactory at a non-trivial scale.

2. Use Conan Lockfiles + Revisions, and a single permanent secondary Conan repository such as “conan-temp” which is only used by CI (not users). This is one of the most important challenges lockfiles address. For each commit or PR, you start with a lockfile (there some options for this step). But, once you have a lockfile, you can rebuild the package you’ve changed and the resulting lockfile will have the new revision. You then upload that revision to the conan-temp repository. As long as all jobs are using lockfiles, the new revision won’t affect any other unrelated jobs. Now, after a rebuild of a library, you will typically need to build downstream consumers of the changed packaged as well, and so you pass the updated lockfile to those jobs. This lockfile will bring in the specific revision you want from the upstream CI job, and will also update the lockfile with the revision of each downstream package which gets rebuilt as a result.

With lockfiles and a single “conan-temp” repository, you can get completely isolated testing of a commit or PR, including the downstream rebuilds and tests. However, be warned that lockfiles are still experimental, and still changing a lot. Lockfiles also bring a number of other implications on the overall workflow, which take time to plan. It is not a drop-in feature, so if you want to change the minimal amount, consider option 1 for now, and perhaps start research and experimentation with lockfiles now.

[memsharded]

The default package_id model for 2.0 is already in place, lets summarize:

• The environment is not part of the package_id. If it is something that fundamentally affects binary compatibility, then it should be modeled into settings or conf.
• The typical use case of environment would be for example to define per machine variability, like the path of a compiler, that can be different in different machines, still exactly the same compiler
• Header-only libraries should not include anything in their package_id, it should always be exactly the same one. Except if for some reason there are headers that are generated differently with different inputs. That should be made most likely a setting or an option of the package.