[bug] buildenv vs runenv for cmake.test() helper.

Follow-up to #12407

I can see now why conan applies [buildenv] to this its cmake.test helper (since it’s really cmake --build <build_folder> --target test), but it feels like many cases with shared libraries would also want the [runenv]. For example, they might need to find .dll files from libraries that were in requires. I would expect most test content to be pretty indifferent to the buildenv (unless doing something unusual, for example testing a header-only library by calling $(CC) -fsyntax-only).

But tests that call executable you just built (which linked to stuff in requires) don’t seem that exotic. So I tested this, and it’s indeed not working. I’m surprised I never hit that for real, but I guess I don’t have all that many shared-library dependencies, and the one that is common (zlib1.dll) is kind of accidentally working because git for windows puts one in PATH, and I mostly live in git-bash.

Environment Details (include every applicable attribute)

• Operating System+version: Windows 10 200H2
• Compiler+version: Visual Studio 2022
• Conan version: 1.53.0
• Python version: 3.10.7

Steps to reproduce (Include if Applicable)

conan create conanfile.py -pr conanprofile.txt

conanfile.py

from conan import ConanFile
from conan.tools.cmake import CMake

class CTestConan(ConanFile):
    name = "ctest"
    version = "1.0"
    settings = "os", "compiler", "build_type", "arch"
    default_options = {"zlib:shared": True}
    generators = "CMakeToolchain", "CMakeDeps"
    requires = "zlib/1.2.12"

    exports_sources = "CMakeLists.txt", "main.cpp"

    def build(self):
        cmake = CMake(self)
        cmake.configure()
        cmake.build()
        cmake.test()
        # self.run("ctest", cwd=self.build_folder, run_environment=True)
        #self.run("ctest", cwd=self.build_folder, env=["conanbuild","conanrun"])

    def package_info(self):
        self.cpp_info.libs = ["hello"]

CMakeLists.txt

cmake_minimum_required(VERSION 3.23)
project(test LANGUAGES CXX)

find_package(ZLIB)

add_executable(main main.cpp)
target_link_libraries(main ZLIB::ZLIB)

enable_testing()
add_test(NAME main COMMAND main)
add_test(NAME zlib1 COMMAND where zlib1.dll)
add_test(NAME env COMMAND ${CMAKE_COMMAND} -E environment)

main.cpp

#include <zlib.h>
int main() {
	z_stream zs = {};
	inflateInit(&zs);
	inflateEnd(&zs);
}

conanprofile.txt

[buildenv]
BUILDENV=1
[runenv]
RUNENV=1
[conf]
tools.env.virtualenv:auto_use=True

Logs (Executed commands with output) (Include/Attach if Applicable)

If run from cmd (with no other zlib1.dll already in PATH from some random place), main fails with Exit code 0xc0000135 (STATUS_DLL_NOT_FOUND) and <build_folder>/Testing/Temporary/LastTest.log shows

1/3 Test: main Exit code 0xc0000135

2/3 Test: zlib1 Command: “C:/Windows/System32/where.exe” “zlib1.dll” Output: INFO: Could not find files for the given pattern(s).

3/3 Test: env Command: “C:/Program Files/CMake/bin/cmake.exe” “-E” “environment” BUILDENV=1

but no RUNENV=1 – so [buildenv] was applied, but not [runenv]. This is what I expected from #12407, but it seems like we would want [runenv] too when running tests.

---

If run from git-bash instead of cmd, all 3 tests “pass”, but the build_folder/Testing/Temporary/LastTest.log shows that this is because we found zlib1 from mingw64:

3/3 Test: zlib1 Command: “C:/Windows/System32/where.exe” “zlib1.dll” Output: C:\Program Files\Git\mingw64\bin\zlib1.dll

And that’s why main.exe succeeded, that was enough to make a simple executable load and close enough to not crash. But we’ve mismatched versions/settings/options pretty much freely, since conan had no part in picking this one.

---

If I replace the cmake.test() with a self.run("ctest", cwd=self.build_folder, run_environment=True), then PATH gets %CONAN_USER_HOME%\.conan\data\zlib\1.2.12\_\_\package\ad5261bf6074807e7189c351b0f79b113bf2f6c0\bin, and so main.exe gets the correct zlib1.dll. But it still doesn’t get the [runenv]RUNENV=1, so apparently run_environment=True still does not pick up the profile’s [runenv]?

If I try to get both by passing a list, i.e.self.run("ctest", cwd=self.build_folder, env=["conanbuild","conanrun"]), I unexpected get neither [buildenv] nor [runenv].

The documentation at https://docs.conan.io/en/latest/reference/conanfile/other.html#output-and-running says this should work

the environment or list of environment activations scripts to pre-pend to the given command.

but the code to force a single str into into a list:

https://github.com/conan-io/conan/blob/5bf9aacedf9cc71c52ef548b079285715b51ba10/conans/model/conan_file.py#L441

will just end up in its else [] case if the caller actually anything other than str

Suggested fix

So it seems like Conanfile.run(env=…) should be fixed to recognize isinstance(list) (as is already documented), and then CMake.test -> CMake._build should pass in env=["conanbuild","conanrun"], rather than just “conanbuild”.

https://github.com/conan-io/conan/blob/5bf9aacedf9cc71c52ef548b079285715b51ba10/conan/tools/cmake/cmake.py#L157-L158 https://github.com/conan-io/conan/blob/5bf9aacedf9cc71c52ef548b079285715b51ba10/conan/tools/cmake/cmake.py#L127

[memsharded]

yeah, I agree with runenv being given priority over (i.e. applied after) buildenv if both are used. Better to give the test executables the libs we meant to be testing, and hope ctest can work with that, rather than vice-versa.

I agree

So, these are the alternatives, to discuss with the team:

• cmake.test() adds both virtualbuildenv and virtualrunenv, with the later higher priority
• cmake.test() uses only virtualrunenv, using internally ctest instead of cmake
• Maybe we could consider explicitly cmake.ctest() as a explicit opt-in into this later behavior, while keeping cmake.test() with the former.

[puetzk]

Also, I’m marking this as look-into for the next release and we will check your suggestion of adding the runenv to the run command by default.

I don’t know that it should be the default for all run commands, but I do think that CMake.test() should probably be adding it (which means passing it along through CMake._build() into ConanFile.run().