Java crashes due to hsperfdata file conflicts across sandboxes

Running bazel build on a fat java/scala project (several thousands of targets) fails when working on linux debian with user namespace enabled.

Issue

Trying to run bazel build with user namespace enabled:

$ sysctl kernel.unprivileged_userns_clone=1

The build runs alright but at some point it crashes with weird memory issue:

ERROR: <target-path>/BUILD:35:1: error executing shell command: '
  rm -rf bazel-out/local-fastbuild/bin/<package>/<target>.jar_temp_resources_dir
  set -e
  mkdir -p bazel-out/local-fastbuild/bin/<target>' failed: Process terminated by signal 6 [sandboxed].
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f094606874b, pid=5, tid=0x00007f09472e0700
#
# JRE version:  (8.0_131-b11) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x96874b]  PerfMemory::alloc(unsigned long)+0x7b
#
# Core dump written. Default location: /home/builduser/.cache/bazel/_bazel_builduser/bc0e462ab01ac9379d22ad058ca1cb1f/bazel-sandbox/4864102460254154064/execroot/__main__/core or core.5
#
# An error report file with more information is saved as:
# /home/builduser/.cache/bazel/_bazel_builduser/bc0e462ab01ac9379d22ad058ca1cb1f/bazel-sandbox/4864102460254154064/execroot/__main__/hs_err_pid5.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

Environment info

The machine is docker container based on debian image

$ uname -a
Linux 167-docker99 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2 (2017-04-30) x86_64 GNU/Linux
builduser@167-docker99:~/ws/bazel-port-isolation$ cat /etc/*-release
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"Bazel version

• The project is fat (several thousands of java / scala targets)
• Bazel was built from a81264e1043dd90e984d9fcef5ce9962dce90d1d
• rules_scala - https://github.com/wix/rules_scala/commit/d66c9d7506ecc6a1b454055b35bd10ef064b9d98 (basically https://github.com/bazelbuild/rules_scala/commit/5d6ff512652b8b55f5d26f6ea69e05d86582d996 with small changes around specs2 versions and test runner env preparation)

additional information

• issue does not happen when unprivileged_userns_clone=0 (but clearly - that’s not a solution)
• with user namespace enabled, bazel 0.5.1 showed this issue . May also be related to #3064 .

[nkoroste]

Any updates on this? I started hitting this issue very often on all linux machines in CI.

My current workaround is to add the following to my .bazelrc

build --enable_platform_specific_config

build:linux --sandbox_tmpfs_path=/tmp

[lberki]

Yeah, in fact, I have to stop myself from campaigning for the flag flip to go into Bazel 6.0 😃

Emotionally, I feel like “Bazel can’t reliably run a JVM in an action” is quite an embarrassing issue, although the fact that this bug has been open for more than five years seems to imply that it’s a less serious issue than my feelings say.