Catkin tool build issues with symlinking

System Info

• Operating System: Linux alexDev-VirtualBox 4.13.0-26-generic #29~16.04.2-Ubuntu SMP Tue Jan 9 22:00:44 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Python Version: Python 2.7.12
• Version of catkin_tools: catkin_tools 0.4.4 © 2014-2018 Open Source Robotics Foundation catkin_tools is released under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)

---

Using Python 2.7.12 (default, Nov 20 2017, 18:23:56) [GCC 5.4.0 20160609]

• ROS Distro: kinetic

Build / Run Issue

• Works with catkin_make
• Works with catkin_make_isolated --merge
• Works with catkin build
• Works with catkin build -p1
• Works with catkin build -v
• Works with VERBOSE=1 catkin build -p1 -v

Expected Behavior

The expected behavior is for all the packages to build successfully and create a tools directory that contains all the executables that we built.

Actual Behavior

When the build is run with either: catkin build or catkin build -p1 the build doesn’t work but when the build is run with catkin build -v then the build successfully builds or others as noted above.

Therefore my main question is, what is the difference between catkin build and catkin build -v where the verbose version of the build works (can build the software successfully) but the non-verbose version doesn’t.

Some background on the error.

1. We are building 4 executables
2. The 4 source files and the make files are copied into the build space
3. Then the CMakeLists.txt will run ‘add_custom_target’ to executable the makefiles pointing to the source in the build directory.
4. The Makefiles will then build the executables to the devel directory.

With the working version: The executables are built in the devel/.private directory and then afterwards it is symlinked to the merged devel space devel/.

With the broken version (catkin build or catkin build -p1): The executables are built in devel/.private directory but fails to symlink them to the merged devel workspace. Instead it throws an error: No handlers could be found for logger "trollius"ed] [fly_tools:make (50%) - 2.3] [shared_interfaces:cmake - 2.2]

And then later it gives the traceback:

[build] An internal error occurred!                                                                                                                                                                        
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/catkin_tools/verbs/catkin_build/build.py", line 549, in build_isolated_workspace
    continue_without_deps=False))
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/executor.py", line 370, in run_until_complete
    return loop.run_until_complete(coroutine)
  File "/home/alex/.local/lib/python2.7/site-packages/trollius/base_events.py", line 350, in run_until_complete
    return future.result()
  File "/home/alex/.local/lib/python2.7/site-packages/trollius/futures.py", line 285, in result
    compat.reraise(type(self._exception), self._exception, exc_tb)
  File "/home/alex/.local/lib/python2.7/site-packages/trollius/tasks.py", line 259, in _step
    result = coro.send(value)
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/executor.py", line 166, in async_job
    stderr=logger.get_stderr_log(),
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/io.py", line 119, in get_stderr_log
    return self._decode(self.stderr_buffer)
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/io.py", line 131, in _decode
    return data.decode('utf-8')
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 3343: invalid start byte

Unfortunately I cannot give the source code to reproduce the error but hopefully there is some information to lead me to help figure out the issue.

1. Do you know why catkin build -v would work and catkin build won’t?
2. The executable is built but it fails to symlink so it seems like a python issue? When i go to the .private directory I can see the generated executable from the makefile even though the build fails
3. Is there any other way to see more verbose catkin build output besides catkin build -v since that makes the build succeed?

[Crowdedlight]

Whats the status on this issue? And when does the fix make it into a release? 😃

I am getting same issue but only on my CI catkin run_tests, locally it works. My CI also success when doing catkin build, but fails consistently on catkin run_tests

[tiusty]

Here is a sequence of buildings:

sequence 1: catkin clean -y catkin build (fails) catkin build (fails) after that

sequence 2: catkin clean -y catkin build -v (works) catkin build (sometimes work, sometimes fails)

sequence 3: catkin clean -y catkin build -v (works) catkin build -v (works) after that

sequence 4: catkin clean -y catkin build (fails) catkin build -v (works) after that

One thing to note: These are the patterns that we experience. Once in a while the catkin build works that doesn’t follow this pattern (like on the first try after a clean) although this is rare. I agree this is probably a race condition issue. I tried to debug the python script by executing it within a debugger. Except there seems to be a lot of parallel processes that make it hard to tell what is going on. Also, a lot of loops occur multiple times and it is hard to break near the time the error occurs.