[question] How can a custom generator find export_sources "origin_folder"?

Question

Is there a way for a generator to obtain the “origin_folder” / originating folder of an exported conanfile.py that’s being processed?

The recipe_folder is close, but may not be under SCM if export_sources is used.

Thanks for your support!

Context

I have a custom generator that attempts to dynamically capture SCM information from custom call to git describe and tools.Git() and generate a git-describe.cmake file with various bits of meta data (e.g. hash, tag, dirty, commits from last tag, etc).

The goal is to drive both Conan’s package version and also CMake’s PROJECT_VERSION from a tag in SCM.

It’s meant to be used in both a package’s set_version and generate methods:

def set_version(self):
    foo = mygenerator(self)
    self.version = foo.version
def generate(self):
    foo = mygenerator(self)
    foo.generate()

I’m attempting to augment the basic approach outlined in the doc How to dynamically define the name and version of a package.

So, the generator would like to know the location where conanfile.py originated from in order to leverage things like tools.Git(folder=conanfile.recipe_folder) on an actual repository in order to obtain git tag information.

The generator works for packages created from source using conan create or if using conan install to install the package requirements in a build folder and build outside of Conan.

However, I’m struggling to get the generator to work with packages that rely on export_sources.

Packages using export_sources fail because the package contents are copied to a temporary folder outside of SCM prior to generating, etc.

This means my custom generator receives a conanfile.py that lives outside of SCM, where quite obviously, tools.Git(folder=conanfile.recipe_folder) can’t be leveraged.

Workarounds

One idea I tried that worked was to have my generator generate the files in set_version and include the generated files alongside the other export_sources

But this seems like a Bad Idea ^TM.

Other Ideas

🏆 conandata.yml

I noticed that the doc for conan.tools.files.update_conandata() has a blurb that sounds eerily similar to what I’m trying to do:

This helper can only be used within the export() method, it can raise otherwise. One application is to capture in the conandata.yml the scm coordinates (like Git remote url and commit), to be able to recover it later in the source() method and have reproducible recipes that can build from sources without actually storing the sources in the recipe.

This is basically what I’m hoping to do. Capture SCM details once. And then later recover them in my generator. And then generate a *.cmake file with what I’ve found.

I’m betting I could smuggle information into my generator using update_conandata and recover it using self.conan_data["sources"]["mydata"].

Perhaps I could either smuggle the “originating” conanfile.py’s path, or whatever Git details I wished to record

I’ve just never seen it suggested to pass information to generators in this way, so I’m a little leery of it.

Edit: Update

Darn. This approach won’t work because conanfile.conan_data seemingly isn’t available inside the generate method. The conandata.yml file is produced with the right key, but I can’t consume it through public Conan API.

Edit: Update 2 Well. This approach will work, but only by convention. I can have my generator look for a conandata.yml and use that if it finds it, then fall back to conanfile.recipe_folder.

May still be preferable because at least leveraging conan_data.yml in this way is supported for some methods. Still less than ideal, so I’m hoping there’s another better solution available.

This approach seems related to: https://github.com/conan-io/conan/issues/10926

Edit: Update 3

Honestly not sure what I was thinking in the first update, conanfile.conan_data is available inside generate.

See https://github.com/conan-io/conan/issues/11717#issuecomment-1198572246 for a proposed resolution.

layout

I’ve also wondered if maybe the new package layout might offer a better way to describe what I’m attempting.

userinfo

By convention, I suppose could probably smuggle the “origin_folder” via user_info.

---

Anyways, thanks for your help and support!

• I’ve read the CONTRIBUTING guide.

[memsharded]

Hi @thorntonryan

I think this is mostly about the process, that needs to run decouple in multiple stages. When packages can be consumed without sources or build at all, just consume the binaries, things need to be prepared for that.

Maybe these ideas can help:

• Whatever needs to be captured from the original source folder, what you git clone, and then conan create, needs to be capture in the export stage. You might use the export() and export_sources() method to extract the information that you need from the Git repo. This could be a useful example: https://docs.conan.io/en/latest/reference/conanfile/tools/scm/git.html#example-implementing-the-scm-feature
• The generate() method runs before the build, and one of the last steps when a conan install happens. This method can recover data from the conan_data, to prepare the build, create build scripts with it, etc. The generate() method responsibility is to prepare the build, and for that, it can do different “generation” tasks:
  • Get information from dependencies, so they can be located by the build, as CMakeDeps do
  • Get some files from the dependencies, maybe copy() them locally if necessary (should not be done if possible to avoid, but sometimes the only way is to copy them). This would be the replacement of the legacy imports() in Conan 2.0
  • Convert Conan configuration like settings and options into something the build system can understand, like CMakeToolchain does.
  • It could be the right place to call an automatic code generation tool that creates source code files (but note that those files do not belong to the “source” stage, but to the “build” stage.

Note that this typically do not include operations over the source of the package itself. It is not the typical place to apply patches (that should be the source() method if the patch is the same for all configurations). Information about the source code, specifically meta-information about the source code, as info that comes from the Git repo containing the source code might not be available here. So it needs to be done earlier in the process, in a 2-step approach.

Please let me know if this clarifies something

[memsharded]

Yes, you are right. Just a clarification, if you are only using it to asign the self.version, this doesn’t need to go to the conandata too. The set_version only executes locally, and at export time. Once it is in the cache, the version self.version will be the correct one, because the set_version() will not be executed and the version will be recovered from the cache metadata instead. So if it is just the version, no need to add it to the conandata.yml at all.

Thanks for your kind words, we love to help! 😃