[feature] Auto generation of cpp info from cmake

• I’ve read the CONTRIBUTING guide.

Motivation

It would be convenient to be ably to cheaply populate cpp info from knowledge that is already contained in cmake build scripts. Better package data with less effort, who doesn’t like that?

def package_info()
	cmake = self._configure_cmake()
	self.cpp_info = cmake.get_cpp_info()

For this to work best, probably needs the sub-components capability that is currently in dev.

All this is just some initial prototyping. Would welcome some outside direction and feedback to help move it along (or discard if there are dead-ends or better ways).

Approach

Rather than trying to go down the rabbit hole of parsing cmake (i.e. package config files). This approach tries to generate information conan needs in easier to consume format.

1. CMake Configure/Generation creates a json file with desired information. This requires some kind of cmake helper function and a template file. Information filled in via configure file to populate variables and file(GENERATE) to convert generator expressions.

2. JSON file is included in package or with recipe

3. Build helper reads JSON and fills in cpp info for you in package_info

Code

Template & Generate Function

{
	"name" : "$<TARGET_PROPERTY:@target@,NAME>",
	"filename" : "$<TARGET_FILE:@target@>",
	"includedirs" :
	{
		"build" : "@BUILD_INCLUDE_DIRS@",
		"install" : "@INSTALL_INCLUDE_DIRS@"
	},
	"cflags" : "$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:@target@,INTERFACE_COMPILE_OPTIONS>;$<TARGET_PROPERTY:@target@,INTERFACE_COMPILE_FLAGS>>",
	"cxxflags" : "$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:@target@,INTERFACE_COMPILE_OPTIONS>;$<TARGET_PROPERTY:@target@,INTERFACE_COMPILE_FLAGS>>",
	"defines" : "$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:@target@,INTERFACE_COMPILE_DEFINITIONS>>",
	"requires" : "$<TARGET_PROPERTY:@target@,INTERFACE_LINK_LIBRARIES>"
}

function(generate_cpp_info)
	foreach(target ${ARGV})
		message(STATUS "Evaluate target: ${target}")
		if (NOT TARGET ${target})
			message(WARNING "Argument \"${target}\" is not a target. Cannot export cpp_info.")
			continue()
		endif()

		get_target_property(INCLUDE_DIRS ${target} INTERFACE_INCLUDE_DIRECTORIES)
		string(REPLACE "$<BUILD_INTERFACE:" "$<1:" BUILD_INCLUDE_DIRS "${INCLUDE_DIRS}")
		string(REPLACE "$<INSTALL_INTERFACE:" "$<0:" BUILD_INCLUDE_DIRS "${BUILD_INCLUDE_DIRS}")
		string(REPLACE "$<BUILD_INTERFACE:" "$<0:" INSTALL_INCLUDE_DIRS "${INCLUDE_DIRS}")
		string(REPLACE "$<INSTALL_INTERFACE:" "$<1:" INSTALL_INCLUDE_DIRS "${INSTALL_INCLUDE_DIRS}")

		find_file(template_file "target-cpp-info-template.json.in")
		if (template_file)
			configure_file(
				${template_file}
				${CMAKE_BINARY_DIR}/target-${target}-cpp-info-template.json
				)
			file(GENERATE
				OUTPUT ${CMAKE_BINARY_DIR}/target-${target}-cpp-info.json
				INPUT ${CMAKE_BINARY_DIR}/target-${target}-cpp-info-template.json)

			# Erase cache variable set by find
			unset(template_file CACHE)
		else()
			message(FATAL_ERROR "Template file not found. Have CMAKE_INCLUDE_PATH "
				"or CMAKE_MODULE_PATH been set to include package directory?")
		endif()
	endforeach()
endfunction()

Package JSON and Populate Cpp Info

The intent would be that most of the logic goes into cmake build helper and it gains a method get_cpp_info, but I made due just adding some code to my recipe.

	  def package(self):
			cmake = self._configure_cmake()
			cmake.install()
			for t in Path().glob('target-*-cpp-info.json'):
				 self.copy(str(t), dst='package-info')

	  def _cmake_targets(self):
			def to_list(cmake_list):
				 return cmake_list.strip(';').split(';')

			for t in Path().glob('package-info/target-*-cpp-info.json'):
				 with open(str(t), 'r') as f:
					  t = json.load(f, object_hook=lambda d: recordtype('target', d.keys())(*d.values()))
					  t.includedirs.build = to_list(t.includedirs.build)
					  t.includedirs.install = to_list(t.includedirs.install)
					  t.cflags = to_list(t.cflags)
					  t.cxxflags = to_list(t.cxxflags)
					  t.defines = to_list(t.defines)
					  t.requires = to_list(t.requires)
					  yield t

	  def package_info(self):
			for t in self._cmake_targets():
				 self.cpp_info.libs.extend([t.name])
				 self.cpp_info.defines.extend(t.defines)
				 self.cpp_info.cflags.extend(t.cflags)
				 self.cpp_info.cxxflags.extend(t.cxxflags)
				 if self.in_local_cache:
					  self.cpp_info.includedirs.extend(t.includedirs.install)
				 else:
					  self.cpp_info.includedirs.extend(t.includedirs.build)

Use in consumer

Consumer has to add call to generate helper in CMakeLists.txt and pass the targets they wish to export. The helper function and file template can be held in a utility package so they don’t need to be added/deployed to consumer. Consumer will generate the json files in the build directory.

add_library(mylibrary ...)
target_compile_option(mylibrary ...)
target_include_directories(mylibrary ...)
target_link_libraries(mylibrary ...)

# helper function must be injected some how, e.g. defined in
# included build script or defined in some module that gets
# included
generate_cpp_info(mylibrary [...])

Application to Poco

I tried applying this to Poco (another cmake project in conan center), to get some feedback with something a bit more real. Would be interested in any other good test projects; the package_info function isn’t really all that big for Poco.

Added my cmake extensions module as a requirement, the parsing code to the recipe, and then the generate call to the CMakeList.txt. I’m leaving out some details of getting the build to work locally just for brevity.

# CMakeListsOriginal.cmake

include(CMakeExtensions)
generate_cpp_info("${Poco_COMPONENTS}")

It generated some target files for the components.

target-Crypto-cpp-info.json
target-Data-cpp-info.json
target-Encodings-cpp-info.json
target-Foundation-cpp-info.json
target-JSON-cpp-info.json
target-MongoDB-cpp-info.json
target-Net-cpp-info.json
target-PocoFoundation-cpp-info.json
target-PocoJSON-cpp-info.json
target-PocoXML-cpp-info.json
target-Redis-cpp-info.json
target-Util-cpp-info.json
target-XML-cpp-info.json
target-Zip-cpp-info.json

Here’s an example of one.

{
	"name" : "Crypto",
	"filename" : "/Users/marianinos/src/conan-center/recipes/poco/all/build_subfolder/lib/libPocoCrypto.a",
	"includedirs" :
	{
		"build" : "/Users/marianinos/src/conan-center/recipes/poco/all/source_subfolder/Crypto/include;",
		"install" : ";include"
	},
	"cflags" : "",
	"cxxflags" : "",
	"defines" : "POCO_STATIC;POCO_NO_AUTOMATIC_LIBS",
	"requires" : "Foundation;/Users/marianinos/.conan/data/openssl/1.0.2s/_/_/package/9c2bc6bb652b363bce80b3b2118be56a4b0fd392/lib/libssl.a;/Users/marianinos/.conan/data/openssl/1.0.2s/_/_/package/9c2bc6bb652b363bce80b3b2118be56a4b0fd392/lib/libcrypto.a"
}

Potential issues

1. For this to work, the information needed by cpp info must exist somewhere in queryable properties that variable expansion or generator expressions can populate. The project must populate target properties correctly, which generally means setting INTERFACE_* properties with the target_* commands.

2. Naming conditioning probably required. I don’t know how the exact requirements of ‘libs’ property, whether it need logical target names or actual filenames. It seems flexible given the Poco recipe manages debug suffixes, but doesn’t necessarily need prefixes or extensions. But some conditioning seems warranted.

3. Haven’t tried multi-config. The file only need to contain information needed for the package though, so it won’t contain Windows settings in a Linux package for example.

TODOs

1. Need to learn more about cpp info model, especially as it might relate to shared libraries and differences between operating systems.

2. Need to track changes coming to cpp info; it looks like it is under heavy dev with sub components, and not sure full impact there.

3. Will need to try out in several environments; I have access to Mac/Linux easy enough. May need to get windows virtual machine or get more familiar with CI if it offers windows test environment.

Other investigated alternatives

1. Python script that parses cmake files. Didn’t seem promising and too much effort. Also information wouldn’t be available until install step.

2. Looked at file api that replaces server mode and is meant to serve IDE integration. There is similar content for targets in there, but I saw at least one issue for cmake/meson integration where Brad King mentioned that it was not a design goal of file api to expose build tree information sufficient to be driven from another build system.

   The template file approach appears to be simpler and can handle things like difference between build/install interfaces, which didn’t seem to exist in file api target content.

Update: Removed links to proto project as they didn’t add much context, and I expect to keep using those repos for other purposes, so links would go stale.

[KerstinKeller]

Hi, I’d like to reopen and revisit this issue.

If there is other way, it also escaped me, but seems unlikely to exists, I’d say it is just too complicated for the value.

Maybe right now it does not matter so much, but the more packages will be used, the more work will have to be put in maintaining all conan center recipes. (Think about the effort to sync recipes with the original repos, possibly patch sources, …). Conan recipes duplicate information which is actually an “outcome” of the recipe (e.g. not an input), and which is available already via the actual build system.

Therefore I think it is a very promising path (longterm) to think about how to extract packaging information from build systems instead of duplicating that information in Conan recipes.

So I tried to do the same approach that @grifcj suggested but also got stuck using the CMake file API, because it did not contain all relevant information. After opening this thread to ask about how to possibly access the necessary information in a parsable format on the CMake discourse forum, this CMake issue has been opened, which also discusses possible C++ package descriptions:

• pkg-config’s .pc files
• The Common Package Specification (CPS)
• A paper from Richard Smith presented to the SG15 C++ committee: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1767r0.html
• libman, and the spec here: https://api.csswg.org/bikeshed/?force=1&url=https://raw.githubusercontent.com/vector-of-bool/libman/develop/data/spec.bs

I think it could possibly be very interesting for Conan to follow approaches or even contribute (from all the experience gained by CCI) to a standardized package description, since Conans cpp_info is basically also a package description.

If at one day, there were a standardized C++ package description, Conan would have to focus a lot less on writing build systems generators (might even be obsolete, if build systems were able to consume such a standardized package description).

[memsharded]

Hi @KerstinKeller

Thanks for your feedback. We totally agree this is the way to go, and we are really looking forward for the community moving in this direction, and we will also certainly contribute as much as possible.

I am not reopening this issue, because the feature request here is clearly different, it request deducing the information from cmake files, which is totally unfeasible for us, and what you are suggesting is actually different. Please submit a new issue for starting a new conversation around this topic.

For a chance of having something useful, it would be needed that the build systems would be willing to generate and consume that information. We could do things, for example generate some intermediate files (the best approach IMO is libman) from the package_info() method. But without the build systems following these files conventions, it will be just another extra layer for us, complicating our life and ours users without any direct benefit. As much as I would like to see build systems agreeing on this, at the moment seems complicated, and out of Conan scope. We will probably try to contribute proposals to C++ SG15 group when possible, that might be a better forum for this.