[bug] conan upload -r fails to resolve `python_requires` from other remotes

Environment Details (include every applicable attribute)

• Conan version: 1.20.2

Steps to reproduce (Include if Applicable)

• create two conan repositories in your local instance of Artifactory server (conan-first and conan-second)
• create a python conan package and upload it to conan-first (let’s call it PyPack)
• create a conan package that python_requires the PyPack package (let’s call it MyPack)
• delete the PyPack from your local ~/.conan/data cache
• upload MyPack to conan-second
• upload fails with following message:

$ conan upload MyPack/5.1.0@user/test -r conan-second
PyPack/3.0.2@microblink/stable: Retrieving from server 'conan-second' 
PyPack/3.0.2@microblink/stable: Trying with 'conan-second'...
ERROR: Error loading conanfile at '/Users/dodo/.conan/data/MyPack/5.1.0/user/test/export/conanfile.py': Unable to find python_requires("PyPack/3.0.2@microblink/stable") in remotes

The problem is that -r conan-second is interpreted as both upload target and as the filter from which python_requires dependencies are to be resolved.

In my use case, I have conan-first as primary conan repository containing stable packages for production use. Now I want my CI to be able to build testing/staging packages and upload them to conan-second, which is a kind of temporary conan repository for testing the code with unreleased packages. This repository has way more administrators that can delete/change already built packages.

[DoDoENT]

Thank you for the hints. I will consider them and see if those fit our workflow (e.g. I will need to find a way to delete temporary artefacts from the “build” repository if some branch fails - I definitely do not want its storage to grow indefinitely nor clean it manually; also to find a way to automatically promote packages when all build branches are OK).

I like the idea of using Artifactory as the database for Jenkins. It has a good value also for non-C++/Conan related stuff. Thank you for that!

[DoDoENT]

It’s not deleted - it never existed in the first place 😃

The reason for this is that I have a separate package build stage in CI which builds conan packages in parallel on different CI nodes for different platforms. However, I do not want any of the builders to upload their packages to the Artifactory server until all builders have successfully built their package. Without that, I would end up with broken packages on my Artifactory server. So, each builder, after it finishes building packages. first calls a dry run of upload to make sure that tgz files appear and then stashes the built packages from the conan cache. Note that only packages that have build folder are stashed, to save servers resources.

When all builders finish, it’s safe to upload the built packages. For that matter Jenkins spawns new N parallel tasks, where each unstashes the stashed conan cache build in previous stage and simply uploads everything to the artifactory. Thus, when building packages, our CI makes sure to upload all built packages, including the packages built by dependency. However, the python_requires package is not built by dependency and thus not stashed on the builder and it’s not so easy to determine all python_requires packages that would need to be stashed - the current workaround is much easier, since the jenkins node always works with empty conan client with temporary conan cache (using Artifactory plugin for Jenkins).

Here is the screenshot of our jenkins process (this job is triggered on every repository after merging PR to stable branch).