generating compile_commands.json for workspace

System Info

• Operating System:

Linux thousandsunny 4.15.0-50-generic #54~16.04.1-Ubuntu SMP Wed May 8 15:55:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Python Version: Python 2.7.12
• Version of catkin_tools:

catkin_tools 0.4.5 (C) 2014-2019 Open Source Robotics Foundation
catkin_tools is released under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)
---
Using Python 2.7.12 (default, Nov 12 2018, 14:36:49) [GCC 5.4.0 20160609]

• Git version: git version 2.7.4
• ROS Distro: kinetic

Build / Run Issue

• Works with catkin_make
• Works with catkin_make_isolated --merge
• Works with catkin build
• Works with catkin build -p1
• I did not read this

Expected Behavior

• catkin build -DCMAKE_EXPORT_COMPILE_COMMANDS=1 should generate a compile_commands.json file inside catkin_ws/build that’s for the entire workspace. I’m using this file for the cquery linter

Actual Behavior

• I’m trying to get a compile_commands.json output with catkin build, the same file that you would get when you use cakin_make
• catkin_make: catkin_make -DCMAKE_EXPORT_COMPILE_COMMANDS=1 would generate a compile_commands.json inside catkin_ws/build which is a nice file that works for the entire workspace.
• catkin build: catkin build -DCMAKE_EXPORT_COMPILE_COMMANDS=1 doesn’t generate a nice file for the entire workspace, but rather individual compile_commands.json for each package. I tried to do a work around by using a python file to concatenate all the compile_commands.json in each subfolder in the build folder. However the generated compile_commands.json doesn’t recognise #include <message_header>.h, where as the compile_commands.json generated through catkin_make does.

Is there a way to properly generate a compile_commands.json for the entire catkin workspace using catkin build?

python script here:

import os
import sys

workspace_path = os.popen('catkin locate --workspace $(pwd)').read().rstrip()

if not workspace_path:
    sys.exit("\033[1;31mNo workspace found in the current directory\033[0m")


ccjson_path = workspace_path+'/compile_commands.json'
target_ccjson_path = workspace_path+'/src'+'/compile_commands.json'

for i in [ccjson_path, target_ccjson_path]:
    if os.path.isfile(i):
        os.system('rm %s' % i)
        print("removed previous compile_commands.json located at %s" % i)

# create a dictionary with file names as keys
# and for each file name the paths where they
# were found
file_paths = {}
for root, dirs, files in os.walk(workspace_path):
    for f in files:
        if f.endswith('compile_commands.json'):
            if f not in file_paths:
                file_paths[f] = []
            file_paths[f].append(root)

# for each file in the dictionary, concatenate
# the content of the files in each directory
# and write the merged content into a file
# with the same name at the top directory
for f, paths in file_paths.items():
    txt = []
    with open(os.path.join(paths[0], f)) as f2:
        txt.append('['+f2.read()[1:-2]+',')
    for p in paths[1:-1]:
        with open(os.path.join(p, f)) as f2:
            txt.append(f2.read()[1:-2]+',')
    with open(os.path.join(paths[-1], f)) as f2:
        txt.append(f2.read()[1:-1]+']')
    with open(f, 'w') as f3:
        f3.write(''.join(txt))

if not os.path.isfile(target_ccjson_path):
    os.system('mv ./compile_commands.json %s' % (target_ccjson_path))


print("\033[1;32mFinished writing compile_commands.json\033[0m")

Steps to Reproduce the Issue

cd ~/catkin_ws
catkin clean
catkin build -DCMAKE_EXPORT_COMPILE_COMMANDS=1

Edit: Made the python script work anywhere in the catkin workspace

[ndepal]

I agree, it would be great if catkin build -DCMAKE_EXPORT_COMPILE_COMMANDS=1 did this.

Based on your python script I made my own shell script that does the same:

#!/bin/sh

cd `catkin locate --workspace $(pwd)`

concatenated="build/compile_commands.json"

echo "[" > $concatenated

first=1
for d in build/*
do
    f="$d/compile_commands.json"

    if test -f "$f"; then
        if [ $first -eq 0 ]; then
            echo "," >> $concatenated
        fi

        cat $f | sed '1d;$d' >> $concatenated
    fi

    first=0
done

echo "]" >> $concatenated

[AlexisTM]

@ndepal @rhklite

My way to merge all compile_commands.json together, using the jq tool to simplify integration with other tools.

sudo apt install jq

jq -s 'map(.[])' build_folder/**/compile_commands.json > compile_commands.json

@LeroyR I agree, catkin_tools should keep compile commands split as it is a split workflow.