Possible deadlock with catkin build

System Info

• Operating System: Ubuntu 14.04 LTS
• Python Version: 2.7
• Version of catkin_tools: 0.4.2
• ROS Distro: Indigo

Build / Run Issue

I apologize in advance for this very imprecise bug report.

I noticed that, starting with the new version 0.4.x, catkin build will hang sometimes. It happens while processing a random package and very rarely (I’d say 3 out of 100 builds on our Jenkins). Also, it will run just fine if I cancel and reschedule the build.

I stumbled upon the --no-install-lock option, which I just added to our build script in the hope that it will resolve this issue. I won’t be able to tell until sufficiently many builds have run, obviously.

In case anyone has an idea where to look for this problem, our build script runs the following commands:

catkin config -w ros --init --no-blacklist --install -j4 -p4 --cmake-args -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_COMPILER=/usr/lib/ccache/gcc -DCMAKE_C_FLAGS_DEBUG="-fmessage-length=0 -Wall -Wextra -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-error=deprecated-declarations -Wno-error=unused-variable -Wno-error=unused-but-set-variable -O0" -DCMAKE_CXX_COMPILER=/usr/lib/ccache/g++ -DCMAKE_CXX_FLAGS_DEBUG="-fmessage-length=0 -Wall -Wextra -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-invalid-offsetof -Wno-unused-local-typedefs -Wno-error=deprecated-declarations -Wno-error=unused-variable -Wno-error=unused-but-set-variable -O0" -DCMAKE_SHARED_LINKER_FLAGS_DEBUG="-Wl,-z,defs" -DCMAKE_EXE_LINKER_FLAGS_DEBUG="-Wl,-z,defs"
catkin clean -w ros --all --yes
catkin build -w ros --verbose --no-status --no-notify --continue-on-failure

The last command outputs:

--------------------------------------------------------------------------------
Profile:                     default
Extending:             [env] /opt/ros/indigo
Workspace:                   /home/jenkins/ros
--------------------------------------------------------------------------------
Source Space:       [exists] /home/jenkins/ros/src
Log Space:         [missing] /home/jenkins/ros/logs
Build Space:        [exists] /home/jenkins/ros/build
Devel Space:        [exists] /home/jenkins/ros/devel
Install Space:     [missing] /home/jenkins/ros/install
DESTDIR:            [unused] None
--------------------------------------------------------------------------------
Devel Space Layout:          merged
Install Space Layout:        merged
--------------------------------------------------------------------------------
Additional CMake Args:       -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_COMPILER=/usr/lib/ccache/gcc -DCMAKE_C_FLAGS_DEBUG=-fmessage-length=0 -Wall -Wextra -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-error=deprecated-declarations -Wno-error=unused-variable -Wno-error=unused-but-set-variable -O0 -DCMAKE_CXX_COMPILER=/usr/lib/ccache/g++ -DCMAKE_CXX_FLAGS_DEBUG=-fmessage-length=0 -Wall -Wextra -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-invalid-offsetof -Wno-unused-local-typedefs -Wno-error=deprecated-declarations -Wno-error=unused-variable -Wno-error=unused-but-set-variable -O0 -DCMAKE_SHARED_LINKER_FLAGS_DEBUG=-Wl,-z,defs -DCMAKE_EXE_LINKER_FLAGS_DEBUG=-Wl,-z,defs
Additional Make Args:        -j4
Additional catkin Make Args: None
Internal Make Job Server:    True
Cache Job Environments:      False
--------------------------------------------------------------------------------
Whitelisted Packages:        None
Blacklisted Packages:        None
--------------------------------------------------------------------------------
Workspace configuration appears valid.

[HannesSommer]

@roehling , thanks for the update. We’ve found a working theory and a dirty workaround that works robustly for us:

Theory:

The catkin internal make job-server runs out of job-tokens (we don’t know why or how; FD 6 and 7 (see above) are relevant, because --jobserver-fds=6,7 is passed on to the child make processes). Since catkin is also using these tokens internally it gets locked indefinitely while trying to get one out of zero available). The child processes get blocked on output because catkins also stops reading from the pipes to the children and their buffer run full at some point.

Workaround:

A script runs periodically that:

1. detects the issue for the catkin processes (very poorly: CPU consumption of it + its children zero for a while …) .
2. For each catkin process ($catkin_pid) having the issue it performs: sudo -u jenkins bash -c "echo -n +++ > /proc/$catkin_pid/fd/7" to inject three fresh job-tokens (catkin seems to use ‘+’ only; one would typically lead to it getting stuck again later).

Given this theory I doubt that this is actually a catkin bug. Interestingly this only happens on (all; Trusty and Xenial) our Jenkins slaves running in Docker containers and only for very big jobs (running > 10min, producing > 10MB verbose output)

[roehling]

Unfortunately, no. The closest thing I found is a warning in the documentation of subprocess.Popen.wait that describes similar symptoms. AFAICT it does not apply here, but maybe I overlooked something.