Better Python code reuse

Currently, it is possible to reuse Python code by requiring it and tools.pythonpath. This is a bit limited because it’s not possible to import the module before the ConanFile is loaded, which makes it impossible to do some things (such as subclassing ConanFile). I tried with a profile as well:

[build_requires]
mypymodule/1.2.3@sztomi/stable

… but this doesn’t work either. Maybe it would make sense for build_requires in profiles to be parsed before the conanfile is loaded. Or a separate, special section just for python deps of the conanfile. Is there a better way? What are your thoughts?

[memsharded]

This has been implemented in https://github.com/conan-io/conan/pull/3337, and will be then available (as experimental feature) in conan 1.7. Please stay tuned for the release, and looking forward your testing and feedback.

[absassi]

@memsharded, I think you misunderstood the functionality of my proposed tools.require(). It does not (and must not) depend at all of the conanfile currently being parsed, since that’s the limitation @sztomi was talking about. The idea is to be able to get parsing dependencies before requiring parsing, so it must not appear inside ConanFile class - and I suggest using pure Python (i.e., a function call from the always-available conans) instead of another file to specify py_requires.

I believe this idea is semantically correct, since these dependencies are not requirements of the package, they’re requirements for the conanfile of the package, and thus they should not be controlled by the contents of ConanFile class, should not be visible or controlled anyway from downstream and they should not view or control any of the upstream requirements of the package (it may do so for the requirements of the parsing requirement it is importing, but that seems overkill for now).

Regarding implementation, it is indeed a complex operation, but the operation is already ready to be used: it is just have to download the package using Conan (if not already in cache) and update sys.path. I believe it is safe to assume the package being required is pure-python and therefore not requiring a build step or anything else, but on the other hand, a full-featured conan install running under the hood might be as easy to implement as just fetching packages. When user calls tools.require("mypymodule/1.2.3@sztomi/stable"), it should:

• call the functions that are equivalent to run conan install mypymodule/1.2.3@sztomi/stable
• obtain the PYTHONPATH from the package_info of the installed package
• add it to sys.path or return a context manager like tools.pythonpath does

Options and settings, if that makes any sense for a Python requirement (C extension modules?), may be specified as extra arguments to tools.require. One little extra care must be taken regarding settings, since it does not make sense to use the same that are being injected to the package: suppose I’m cross-compiling some C package and its conanfile.py requires a Python package with a C extension, I want to build that extension to run in my machine, not on the package’s target. To solve this, my suggestion is to simplify: for a first version of tools.require do not make options available and forbid installing packages that specify any settings. This is OK for all real use cases - we are only limiting the way one can specify how to build extension modules for reusable conanfiles. Or perhaps disable building at all in this case.

Thanks