`conan create` doesn't always get a fresh copy of the source

I had modified the conan recipe for googletest to fetch a branch of my own fork of googletest, and tried to compile it with conan create. The build failed because of C++ syntax errors on the code I had changed, so I changed the source, commited the changes, and pushed the code.

However, when running conan create again to generate a package from my modified version of googletest, I realized that it wasn’t fetching the newest sources. Of course, I wasn’t using --keep-source nor --keep-build. This is what I tried:

• changing the sources on the git repo again → didn’t work
• changing the timestamp of the recipe by editing it, adding a character then removing it, then saving the file → didn’t work
• adding some dummy code to the recipe so it has a new checksum → worked !

This is a problem. Even if the recipe didn’t change, it may depend on a repository whose code has changed. By not using any --keep-* option, I expect the source to be cloned everytime, not just when the recipe changes, otherwise it completely breaks a continuous integration flow.

Using Conan 1.4.4 on Windows 2016 Server

• I’ve read the CONTRIBUTING guide.
• I’ve specified the Conan version, operating system version and any tool that can be relevant.
• I’ve explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[liberforce]

The --keep-source is to not remove the sources even when the recipe is changing, and doing the export.

 -k, -ks, --keep-source
                        Do not remove the source folder in local cache. Use
                        this for testing purposes only

This doesn’t mention changes in the recipe. The good semantics behind --keep-source that were self-explanatory now feel clunky with such exceptions to the rule. You should not have to think about the implementation to know how something works, the interface should be enough.

If you don’t need the “caching” of the source, the answer is very simple.

I need the caching too. The caching makes sense when fetching a tarball or a tag of a big library, like boost. It’s just that when I need the caching, I can use --keep-source, whereas currently I’d need to delete the source by hand.

Do the thing directly in the build() method.

Again, getting sources in something called build(), when there’s already a method called source to get the sources feels wrong. I know we already had this talk about the unit tests, but if build() is about making the package, then you have a source() method because those are different things.

Words have a meaning, and if you change their meaning to fit some other purpose, it increases the learning curve as it paves the way for incomprehension between people trying to communicate on how to use your product. I know I may look annoying and nitpicking on this stuff, option names, method names, etc., but that’s because it helps making a tool that is easy to use.

This is how you think things should be. Many other users are doing frequent changes to their recipes, while their create their recipes, and not touching the code at all.

Sure, that’s a frequent use case too. But when doing CI, you also need to produce packages when the source changed and the recipe did not. You need to perform incremental builds whenever possible, because you want the latest version of every component. You need to follow reverse dependencies to rebuild the whole stack if a low level package changed.

Sure, I could pre-process the recipe to change the commit hash when the source has changed, leading to source() getting the source for me, but that looks ugly to me, as conan already knows about the source, and the source() method has so much flexibility that it would be a PITA to make sure all the recipes are written the right way for this to work.

Maybe, if you are doing several builds and rebuilds in the local cache, with sources changes, etc., that could mean that you need to use the developer flow instead.

Adding @danimtb here. About the “developer flow”, the main flow advertised in the documentation is called the local development flow, I suppose that’s what you’re talking about.

Now the problem is that it gives a link to the bincrafters blog, and I had already read their articles about the packaging flows. Their last blog post about it talks about the Conan Package Flow. Which uses conan create all the way, and recommends that for out-of source packaging, which is what I started doing after reading that post.

But this seems in contradiction with what the documentation says:

The local workflow encourages users to do trial-and-error in a local sub-directory relative to their recipe

What I understand is that the bincrafters changed their workflow, then changed back. Both workflows are very different, however conan documentation only link to the root of the blog, making the different flows and when using them is relevant unclear. I chose the one fitting out-of-tree, because I’m just evaluating conan for now, and that seamed less intrusive.

I suppose what you call “developer flow” is what is explained in the rest of that documentation page, and corresponds to what bincrafters advertised for in-source packaging. I’ll see if that fits my needs. I’m not sure however, as I wanted to use the local repository as a cache to make incremental builds easier.

Last remark about the worflows: giving them a formal name, sharing that with the bincrafters, and using the names consistently would make things clearer. I may not be the only one having a hard time taming the beast on using Conan with CI.

[memsharded]

Hi @AndreyMlashkin

Let me clarify: Use of settings and options in the source() method is deprecated. CONAN_V2_MODE will raise an error for them. It will soon be removed. Whatever is in the source() method should be completely generic to all different settings, options and configurations. Whatever is specific for a branch or configuration should be managed in the build() method.

It is clearly explained in the documentation with a note highlighting it: https://docs.conan.io/en/latest/reference/conanfile/methods.html#source