Development sources in project

Visual Studio project files, and likely others, can potentially include sources and PDBs for dependencies. While some packages have PDBs, none have sources. This is understandable, since typically if you are building a package you do not need sources or documentation, but for people using conan packages during development the requirements are different. I am a fan of #1377, which helps in this regard. It is also ideal to have the sources automatically linked to, in this case, the VS Project file after running conan build.

One example is that I had to debug an application which used Qt, and the package I was using was from a repository rather than a local build where I could grab the source. If there was a source component linked to the same package version I could optionally download as a developer and link into the build projects or use a generator to make a text file with the source directory, it would definitely improve my quality of life and efficiency as a developer. I understand that C++ has always had issues with things like this, especially on windows, and that Conan is novel, but other languages often already have solved this in various ways. Obviously, no alternative exists that can do this for C++ on Windows, but it is definitely a concerning issue from my perspective.

TL;DR

• Need components which live on repositories that are linked to their versioned package counterparts.
• Need source component which provides the exact source used to produce the package for debugging purposes.
• Need a generator to output the source directories of all dependencies to a text file.
• Need to hook the source directories, if used, into Visual Studio project files and others if the capability exists.

[olivren]

Here are some helper functions that I include in all my recipes. The usage is very simple: just invoke patch_pdb(self) in the package method or your recipe. In your project’s conanfile, you should import all the pdb files, so that they end up located next to their respective dlls. If the sources of a Conan package exist in the standard location (~/.conan/data/MyLib/1.0/me/stable/source), then Visual Studio will be able to step into the library code showing the source files.

This code relies on the Microsoft tools pdbstr.exe and srctool.exe.

I’d like to do the same for GDB at some point. It seems that this could be achieved by creating a .gdbinit file, contaning multiple set substitute-path commands.

def find_pdbs(base_dir):
    '''
    Iterate on all pdb files in `base_dir`, yielding their full path (start with `base_dir`)
    '''
    for root, dirs, files in os.walk(base_dir):
        for pdb in fnmatch.filter(files, '*.pdb'):
            yield os.path.join(base_dir, root, pdb)

# This is the `srcsrv` entry that will be written in Pdb files. This entry
# is interpreted by the debuggers in order to find, on the local machine,
# the files referenced in the Pdb file as absolute files on the compilation
# machine.
#
# The documentation of this entry can be found in the file `srcsrv.doc` of
# the package Debugging Tools for Windows. Basically, the compiler expects
# to fin the file at the location described by `SRCSRVTRG`. If it is not,
# then the debugger runs `SRCSRVCMD`, who is responsible to write the file
# to `SRCSRVTRG`. We do that using a python script, that copies a file it
# searches in the Conan's source directory of the package. This entry must
# list all the files that the Pdb references that we want to map, using the
# syntax absolutepath*something*otherthing. "absolutepath" must be exactly
# the path referenced in the Pdb (as given by `srctool`), and the rest can`
# contain anything. For the line abc*def, "abc" is assigned to %var1% and
# "def" is assigned to %var2%.
#
_srcsrv_template = """SRCSRV: ini ------------------------------------------------
VERSION=1
SRCSRV: variables ------------------------------------------
CONANPKGNAME={pkgname}
CONANPKGVERSION={pkgversion}
CONANPKGUSER={pkguser}
CONANPKGCHANNEL={pkgchannel}
CONANPKG=%CONANPKGNAME%/%CONANPKGVERSION%@%CONANPKGUSER%/%CONANPKGCHANNEL%
SRCSRVTRG=%targ%\%CONANPKGNAME%\%CONANPKGVERSION%\%CONANPKGUSER%\%CONANPKGCHANNEL%\%var2%
SRCSRVCMD=python -c "import subprocess, shutil, sys; root=subprocess.check_output(['conan', 'info', '--paths', '--only', 'source_folder', '%conanpkg%']).splitlines()[1][19:]; shutil.copy(root + r'\%var2%', sys.argv[1])" %srcsrvtrg%
SRCSRV: source files ---------------------------------------
{files}
SRCSRV: end ------------------------------------------------
"""

def _paths_to_srcsrv_source_section(paths, sources_dir):
    '''
    Convert a list of file paths referenced in a Pdb, to the "source files" section
    of the srcsrv section for this Pdb. Each line is composed of the full referenced
    path, a star, and the same path relative to `sources_dir`. Paths that are outside
    of `sources_dir` are ignored.
    '''
    lines = []
    for path in paths:
        if path.startswith(sources_dir):
            lines.append('{}*{}'.format(path, os.path.relpath(path, sources_dir)))
    return '\n'.join(lines)

def patch_pdb(conanfile):
    '''
    Find all the Pdb files in the package dir of the conanfile, and path them.
    The patch consist of adding a special entry `srcsrv` in the Pdb, using the Microsoft
    tool `pdbstr`. This entry contains the instructions for the debugger on how to map
    file names referenced in the Pdb to a local file.
    '''
    
    # The source files that will be referenced in the Pdb can come from the
    # Conan "source" or "build" directory, depending on the `no_copy_source`
    # attribute. The path is lowercased, because the Pdb format stores all
    # paths as lowercase apparently.
    if hasattr(conanfile, 'no_copy_source') and conanfile.no_copy_source:
        sources_dir = conanfile.source_folder.lower()
    else:
        sources_dir = conanfile.build_folder.lower()
    module = os.path.dirname(__file__)
    srctool = '{}\\bin\\srctool.exe'.format(module)
    pdbstr = '{}\\bin\\pdbstr.exe'.format(module)
    for pdb in find_pdbs(conanfile.package_folder):
        refs = subprocess.check_output([srctool, '-r', pdb]).splitlines()
        srcsrv_sources = _paths_to_srcsrv_source_section(refs, sources_dir)
        srcsrv = _srcsrv_template.format(pkgname=conanfile.name,
                                         pkgversion=conanfile.version,
                                         pkguser=conanfile.user,
                                         pkgchannel=conanfile.channel,
                                         files=srcsrv_sources)
        fd, path = tempfile.mkstemp()
        f = os.fdopen(fd, 'w')
        f.write(srcsrv)
        f.close()
        subprocess.call([pdbstr, '-w', '-p:{}'.format(pdb), '-s:srcsrv', '-i:{}'.format(path)])
        os.remove(path)

[olivren]

No, sorry for the confusion. This quote was not about your comment, but about the original request by @vadixidav, or at least as I understand it. My comprehension of his proposal is the introduction of a “source package” alongside the existing “binary package”, that could be optionally installed by a developer (“If there was a source component linked to the same package version I could optionally download as a developer…”). This “source package” would certainly be created by a recipe method similar to package, maybe package_sources, that would produce an independant artifact uploadable to a Conan server.

I tried to understand the proposal in your comment, reading the content of the zip, but I fail to understand the approach. I don’t know anything to CMake, so maybe that’s the problem 😃