Support for sandboxed nix builds

I’m trying to get this plugin to work with a nix build, however, having issues with the jvm plugins being able to be ran:

[debug] Executing protoc with [--plugin=protoc-gen-jvm_0=/tmp/protocbridge9859361803622670227, --jvm_0_out=/build/mantis/target/scala-2.13/src_managed/main, --jvm_0_opt=grpc, -I/build/mantis/src/main/protobuf, /build/mantis/src/main/protobuf/extvm/msg.proto] and extraEnv=Vector((SCALAPB_SECONDARY_OUTPUT_DIR,/tmp/protocbridge-secondary9760907634468689002))
/tmp/protocbridge9859361803622670227: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--jvm_0_out: protoc-gen-jvm_0: Plugin failed with status code 1.
[error] java.lang.RuntimeException: protoc returned exit code: 1

I’m able to compile the build fine with sbt compile and nix-shell. However, this looks to be a supported usecase.

Additional Info

Using sbt-derivation to bootstrap the sbt dependencies

sbt command being ran:

# Executing command line:
/nix/store/545lbvmxildy1j8ijv2l0xc4flaq4q9a-openjdk-15.0.1-ga/lib/openjdk/bin/java
-Dfile.encoding=UTF-8
-Dsbt.boot.directory=.nix/boot
-Dsbt.global.base=.nix/base
-Dsbt.ivy.home=.nix/ivy
-Dnix=true
-Xms1024m
-Xmx1024m
-Xss4M
-XX:ReservedCodeCacheSize=128m
-Dscala.ext.dirs=.nix/base/java9-rt-ext-n_a_15_0_1_internal
-jar
/nix/store/mja02w4d8y00fbqs8id7rzrv0jcmzpny-sbt-1.4.6/share/sbt/bin/sbt-launch.jar
-debug
compile

command-line command being ran:

running "export PATH=$PATH:/nix/store/yazg2m0y3dm15fcllg4aswppvcy364dl-git-2.29.2/bin:/nix/store/ylp0sjddbrx0m4djv9ljxap204ynla1n-protobuf-3.14.0/bin:/nix/store/545lbvmxildy1j8ijv2l0xc4flaq4q9a-openjdk-15.0.1-ga/bin; JAVA_HOME=/nix/store/fwzqlr3jcg51wm6ds62fi9g2qjd6i58k-openjdk-15.0.1-ga-jre/lib;  sbt compile --debug -Dnix=true" to warm up the caches

Even patching the build.sbt to reference the nixstore path for PB.protocExecutable

  ...
  root.settings(PB.protocExecutable := file("/nix/store/ylp0sjddbrx0m4djv9ljxap204ynla1n-protobuf-3.14.0/bin/protoc"))

Thanks for the initial nix-shell though 😃

[thesamet]

@jonringer , I released a new version of sbt-protoc (1.0.0-RC7-1) which hardcodes /bin/sh instead of /usr/bin/env sh, and also allows overriding the shell used with PROTOCBRIDGE_SHELL environment variable. Since sandboxed environments do provide /bin/sh then this variable isn’t necessary for this.

In addition, sbt-protoc downloads protoc for you, there’s no need to use nix’s provided protobuf, nor a system property for nix, and so. coreutils is needed since protocbridge relies on cat. Another issue is that the default download directory is in the home directory, so we need to set PROTOC_CACHE to some other dir. Finally, I had to comment out in build.sbt few things that were causing failures in a way that’s irrelevant for ScalaPB.

Here are the files that work for me: matris.nix:

let
  sbt-derivation = import (builtins.fetchTarball {
    url =
      "https://github.com/zaninime/sbt-derivation/archive/9666b2b589ed68823fff1cefa4cd8a8ab54956c1.tar.gz";
    sha256 = "17r74avh4i3llxbskfjhvbys3avqb2f26pzydcdkd8a9k204rg9z";
  });
in { pkgs ? import <nixpkgs> { overlays = [ sbt-derivation ]; } }:

let
  nativeBuildInputs = with pkgs; [ git openjdk coreutils ];
  binPath = pkgs.lib.makeBinPath nativeBuildInputs;
in
pkgs.sbt.mkDerivation rec {
  pname = "mantis";
  version = "1.0.0-dev";

  depsSha256 = "0000000000000000000000000000000000000000000000000000";

  #src = /home/jon/work/mantis;
  src = pkgs.fetchFromGitHub {
    owner = "input-output-hk";
    repo = "mantis";
    rev = "682edf560f6fb26b6e8e6aa2ddad066e85f615c7";
    sha256 = "sha256-AfsIItSIrpynIoewYllqtOmNMyiOZk62iIRkrEWJG3M=";
    fetchSubmodules = true;
  };

  patches = [
    (pkgs.substituteAll {
      src = ./protobuf.patch;
      inherit (pkgs) protobuf;
    })
  ];

  depsWarmupCommand = "export PROTOC_CACHE=.nix/protoc-cache; export PATH=$PATH:${binPath}; sbt compile --debug";

  buildPhase = ''
    sbt assembly
  '';

  installPhase = ''
    sbt stage
    install -D target/universal/stage/ -t $out
  '';
}

protobuf.patch:

diff --git a/build.sbt b/build.sbt
index f2f11869a..70a08eb29 100644
--- a/build.sbt
+++ b/build.sbt
@@ -70,11 +70,6 @@ val root = {
     .settings(inConfig(Ets)(Defaults.testSettings): _*)
     .settings(inConfig(Snappy)(Defaults.testSettings): _*)
     .settings(inConfig(Rpc)(Defaults.testSettings): _*)
-
-  if (!nixBuild)
-    root
-  else
-    root.settings(PB.protocExecutable := file("protoc"))
 }

 scalacOptions := Seq(
@@ -105,11 +100,11 @@ PB.targets in Compile := Seq(
 // have the protobuf API version file as a resource
 unmanagedResourceDirectories in Compile += baseDirectory.value / "src" / "main" / "protobuf"

-(test in Evm) := (test in Evm).dependsOn(solidityCompile).value
+// (test in Evm) := (test in Evm).dependsOn(solidityCompile).value
 (sourceDirectory in Evm) := baseDirectory.value / "src" / "evmTest"

-(scalastyleConfig in Test) := baseDirectory.value / "scalastyle-test-config.xml"
-scalastyleSources in Test ++= { (unmanagedSourceDirectories in Integration).value }
+// (scalastyleConfig in Test) := baseDirectory.value / "scalastyle-test-config.xml"
+// scalastyleSources in Test ++= { (unmanagedSourceDirectories in Integration).value }

 // Packaging
 mainClass in Compile := Some("io.iohk.ethereum.App")
diff --git a/project/plugins.sbt b/project/plugins.sbt
index 5f09c46cc..8d3bafe0e 100644
--- a/project/plugins.sbt
+++ b/project/plugins.sbt
@@ -3,7 +3,7 @@ addSbtPlugin("org.scoverage" % "sbt-scoverage" % "1.6.1")
 addSbtPlugin("org.scalastyle" %% "scalastyle-sbt-plugin" % "1.0.0")
 addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.7.5")
 addSbtPlugin("com.thoughtworks.sbt-api-mappings" % "sbt-api-mappings" % "3.0.0")
-addSbtPlugin("com.thesamet" % "sbt-protoc" % "1.0.0-RC7")
+addSbtPlugin("com.thesamet" % "sbt-protoc" % "1.0.0-RC7-1")
 addSbtPlugin("com.timushev.sbt" % "sbt-updates" % "0.5.1")

 libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.10.9"

[jonringer]

Thanks for the incredibly fast turn around on this. I appreciate it greatly! 😃