Protoc import errors

I am failing to generate scala sources from pre-defined proto files, using Mu version: 0.18.0

The proto files are sourced from JARs published by the OpenCensus project, and my sbt file is configured as follows:

lazy val ocensusProto = "io.opencensus"    % "opencensus-proto" % "0.2.0"
lazy val muFs2 = "io.higherkindness"       %% "mu-rpc-fs2"      % "0.18.0"
lazy val muRpcServer = "io.higherkindness" %% "mu-rpc-server"   % "0.18.0"

lazy val protocol = project
  .in(file("modules/protocol"))
  .configure(BuildSettings.profile)
  .settings(name := "apm-collector-protocol", libraryDependencies ++= Seq(muRpcServer, ocensusProto, scalaTest % Test))
  .settings(
    Seq(
      idlType := "proto",
      srcGenSerializationType := "Protobuf",
      srcGenJarNames := Seq("opencensus-proto"),
      srcGenTargetDir := (Compile / sourceManaged).value / "compiled_proto",
      libraryDependencies ++= Seq("io.higherkindness" %% "mu-rpc-channel" % Versions.mu),
      sourceGenerators in Compile += (Compile / srcGen).taskValue
    )
  )

I also have the idl-gen plugin configured as follows in plugins.sbt:

addSbtPlugin("io.higherkindness" % "sbt-mu-idlgen"            % "0.18.0")

I have been scanning the sbt plugin code to parse protocol buffers files, and I see that it tries to resolve import statements that are then passed as arguments to protoc. However, when I compile it appears that the imports are not resolved correctly. The output is as follows:

sbt:apm-collector-protocol> compile
[info] Updating ...
[info] Done updating.
protoc-jar: protoc version: 3.6.0, detected platform: osx-x86_64 (mac os x/x86_64)
protoc-jar: embedded: bin/3.6.0/protoc-3.6.0-osx-x86_64.exe
protoc-jar: executing: [/var/folders/m3/_bxmtk_13z1fnm54cczf9w5h0000gn/T/protocjar3473633784860612324/bin/protoc.exe, --plugin=protoc-gen-proto2_to_proto3, --include_imports, --descriptor_set_out=trace_config.proto.desc, --proto_path=/Users/davidc/work/apm-collector/modules/protocol/target/scala-2.12/resource_managed/main/proto/opencensus/proto/trace/v1, trace_config.proto]
protoc-jar: protoc version: 3.6.0, detected platform: osx-x86_64 (mac os x/x86_64)
protoc-jar: embedded: bin/3.6.0/protoc-3.6.0-osx-x86_64.exe
protoc-jar: executing: [/var/folders/m3/_bxmtk_13z1fnm54cczf9w5h0000gn/T/protocjar16595508427019335140/bin/protoc.exe, --plugin=protoc-gen-proto2_to_proto3, --include_imports, --descriptor_set_out=trace.proto.desc, --proto_path=/Users/davidc/work/apm-collector/modules/protocol/target/scala-2.12/resource_managed/main/proto/opencensus/proto/trace/v1, trace.proto]
opencensus/proto/resource/v1/resource.proto: File not found.
google/protobuf/timestamp.proto: File not found.
google/protobuf/wrappers.proto: File not found.
trace.proto: Import "opencensus/proto/resource/v1/resource.proto" was not found or had errors.
trace.proto: Import "google/protobuf/timestamp.proto" was not found or had errors.
trace.proto: Import "google/protobuf/wrappers.proto" was not found or had errors.
trace.proto:173:5: "google.protobuf.Timestamp" is not defined.
trace.proto:133:3: "google.protobuf.Timestamp" is not defined.
trace.proto:144:3: "google.protobuf.Timestamp" is not defined.
trace.proto:297:3: "opencensus.proto.resource.v1.Resource" is not defined.
trace.proto:304:3: "google.protobuf.BoolValue" is not defined.
trace.proto:308:3: "google.protobuf.UInt32Value" is not defined.
[error] higherkindness.skeuomorph.ProtobufCompilationException: Protoc failed to compile protobuf file
[error]         at higherkindness.skeuomorph.protobuf.ParseProto$.$anonfun$runProtoc$2(ParseProto.scala:65)
[error]         at cats.MonadError.$anonfun$ensure$1(MonadError.scala:14)
[error]         at cats.effect.internals.IORunLoop$.liftedTree3$1(IORunLoop.scala:225)
[error]         at cats.effect.internals.IORunLoop$.step(IORunLoop.scala:225)
[error]         at cats.effect.IO.unsafeRunTimed(IO.scala:321)
[error]         at cats.effect.IO.unsafeRunSync(IO.scala:240)
[error]         at higherkindness.mu.rpc.idlgen.proto.ProtoSrcGenerator$.generateFrom(ProtoSrcGenerator.scala:46)
[error]         at higherkindness.mu.rpc.idlgen.Generator.$anonfun$generateFrom$1(Generator.scala:30)
[error]         at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)
[error]         at scala.collection.Iterator.foreach(Iterator.scala:937)
[error]         at scala.collection.Iterator.foreach$(Iterator.scala:937)
[error]         at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
[error]         at scala.collection.IterableLike.foreach(IterableLike.scala:70)
[error]         at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
[error]         at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
[error]         at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)
[error]         at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)
[error]         at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
[error]         at higherkindness.mu.rpc.idlgen.Generator.generateFrom(Generator.scala:29)
[error]         at higherkindness.mu.rpc.idlgen.Generator.generateFrom$(Generator.scala:25)
[error]         at higherkindness.mu.rpc.idlgen.proto.ProtoSrcGenerator$.generateFrom(ProtoSrcGenerator.scala:33)
[error]         at higherkindness.mu.rpc.idlgen.GeneratorApplication.generateFrom(GeneratorApplication.scala:63)
[error]         at higherkindness.mu.rpc.idlgen.IdlGenPlugin$.$anonfun$idlGenTask$1(IdlGenPlugin.scala:186)
[error]         at sbt.util.FileFunction$.$anonfun$cached$1(FileFunction.scala:73)
[error]         at sbt.util.FileFunction$.$anonfun$cached$4(FileFunction.scala:147)
[error]         at sbt.util.Difference.apply(Tracked.scala:313)
[error]         at sbt.util.Difference.apply(Tracked.scala:293)
[error]         at sbt.util.FileFunction$.$anonfun$cached$3(FileFunction.scala:143)
[error]         at sbt.util.Difference.apply(Tracked.scala:313)
[error]         at sbt.util.Difference.apply(Tracked.scala:288)
[error]         at sbt.util.FileFunction$.$anonfun$cached$2(FileFunction.scala:142)
[error]         at higherkindness.mu.rpc.idlgen.IdlGenPlugin$.$anonfun$taskSettings$6(IdlGenPlugin.scala:159)
[error]         at scala.Function1.$anonfun$compose$1(Function1.scala:44)
[error]         at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:40)
[error]         at sbt.std.Transform$$anon$4.work(System.scala:67)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:269)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error]         at sbt.Execute.work(Execute.scala:278)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:269)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:178)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:37)
[error]         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error]         at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[error]         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error]         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error]         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error]         at java.base/java.lang.Thread.run(Thread.java:835)
[error] (srcGen) higherkindness.skeuomorph.ProtobufCompilationException: Protoc failed to compile protobuf file
[error] Total time: 2 s, completed 28 May 2019, 15:06:22
sbt:apm-collector-protocol> 

Is there a way that I can specify dependency ordering, and also, what is the best way to include the googlle protocol buffer timestamp dependency? I have tried including the google protobuf project as a JAR to srcGen, but it also fails to compile due to import issues.

I would greatly appreciate any input as to how I can get this to compile, I am willing to create a demo project to reproduce if necessary.

Thank you for your help.

[cb372]

This issue should be fixed in the latest version (0.20.1) of mu-scala. I’ve confirmed that the mu-opencensus repo kindly provided by @david-cromberge now compiles successfully.

As an example of a complex real-world protobuf defintion, that repo uncovered quite a few different bugs in our codegen feature, so it’s been really helpful. Thank you!

[ghost]

@cb372 @bilki that is outstanding - it works like a charm! I’m glad I was able to assist with a complex example. Thank you for making the required improvements and fixes 🎉