ImportError: No module named catkin.environment_cache

We’re getting an intermittent issue affecting our CI builds. It pops up at random on different packages, but always takes this form:

00:12:49.053   File "/tmp/buildd/workspace/debian/tmp/build/<pkg>/catkin_generated/generate_cached_setup.py", line 20, in <module>
00:12:49.053     from catkin.environment_cache import generate_environment_script
00:12:49.053 ImportError: No module named catkin.environment_cache

This is running inside a cowbuilder, on vmware, with no additional containerization or isolation. The overall build is a large parallel affair being managed by catkin_tools. I’ve reported this there too (https://github.com/catkin/catkin_tools/issues/378), but @jbohren suggests (and I’m inclined to agree) that this may only be fixable within catkin itself.

Looking at the templated logic here, I’m wondering if we’re looking at a race condition with the copy-on-write filesystem?

It’s not clear to me where an additional sync call is required, or even how to reliably reproduce this problem, but I’d be delighted to accept counsel on either of those points. In the meantime, would we consider a workaround which, say, pauses for 100ms and then retries the whole import/try-block in the event of the second import failing?

[mikepurvis]

For any future travellers, this has been confirmed as a catkin_tools bug around inadequate filesystem mutexing for large parallel builds. The fix is in https://github.com/catkin/catkin_tools/pull/391.

[dirk-thomas]

what should I do?

@guo1104b Ask your question on answers.ros.org instead of on a closed ticket with a different topic.