Bazel branch coverage ignores Jacoco filtering

Description of the problem / feature request:

Bazel counts branch coverage also for code that would be filtered out by Jacoco. Function and line coverage (when looking at reports generated by genhtml) seem to be respecting Jacoco filtering.

Jacoco is constantly improving filtering (https://www.jacoco.org/jacoco/trunk/doc/changes.html), but these changes (e.g. filtering out generated bytecode for Java and Kotlin) aren’t then taken by Bazel branch coverage.

Bugs: what’s the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

1. Check out https://github.com/gergelyfabian/bazel-scala-example/tree/jacoco_coverage_fix.
2. In bazel repo: bazel build src/java_tools/junitrunner/java/com/google/testing/coverage:JacocoCoverage_jarjar_deploy.jar
3. Copy bazel’s bazel-bin/src/java_tools/junitrunner/java/com/google/testing/coverage/JacocoCoverage_jarjar_deploy.jar to bazel-scala-example’s tools/
4. Run tools/coverage.sh
5. You’ll see in the coverage report branch coverage for lines that are not covered by line coverage (example-lib/src/main/scala/mypackage/Foo.scala):

      23         [ +  + ]:            :       d <- if (foo.a < 10) {
      24                 :            :         List(4)
      25                 :            :       } else {
      26                 :            :         List(14)
      27                 :            :       }

1. Branch coverage is 5/30 for example-lib/src/main/scala/mypackage/Foo.scala.

Note: These lines are not covered because of a bug in Jacoco 0.8.3 (that Bazel currently uses), it was fixed in Jacoco 0.8.5.

When in contrast I follow the instructions to upgrade Jacoco in https://github.com/bazelbuild/bazel/issues/11674#issuecomment-739725257 (to Jacoco 0.8.3 with the backported Scala 2.12 lambda fix) and then run the above steps the line coverage changes (the above lines 23-27 are now covered) and the branch coverage stays the same for example-lib/src/main/scala/mypackage/Foo.scala (5/30).

Compare with plain Jacoco:

1. Download http://search.maven.org/remotecontent?filepath=org/jacoco/jacoco/0.8.5/jacoco-0.8.5.zip and http://search.maven.org/remotecontent?filepath=org/jacoco/jacoco/0.8.3/jacoco-0.8.3.zip.
2. Unpack them into /tmp/jacoco_0.8.3 and /tmp/jacoco_0.8.5.
3. Install Scala 2.12.12 into ~/opt/scala-2.12.12
4. Copy bazel-scala-example/example-lib/src/main/scala/mypackage/Foo.scala to /tmp/src/mypackage/Foo.scala
5. cd /tmp
6. ~/opt/scala-2.12.12/bin/scalac src/mypackage/Foo.scala -d classes
7. cd /tmp/jacoco_0.8.3
8. java -javaagent:lib/jacocoagent.jar -cp ~/opt/scala-2.12.12/lib/scala-library.jar:…/classes mypackage.Foo
9. java -jar lib/jacococli.jar report jacoco.exec --classfiles …/classes --sourcefiles …/src --html report
10. cd /tmp/jacoco_0.8.5
11. java -javaagent:lib/jacocoagent.jar -cp ~/opt/scala-2.12.12/lib/scala-library.jar:…/classes mypackage.Foo
12. java -jar lib/jacococli.jar report jacoco.exec --classfiles …/classes --sourcefiles …/src --html report
13. Compare the reports for Jacoco 0.8.3 and 0.8.5. You’ll see that branch coverage is changing and that for 0.8.3 lines that have been filtered aren’t counted as branches hit or total.

What operating system are you running Bazel on?

Ubuntu 20.04.

What’s the output of bazel info release?

release 3.7.1

If bazel info release returns “development version” or “(@non-git)”, tell us how you built Bazel.

Replace this line with your answer.

What’s the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://github.com/gergelyfabian/bazel-scala-example 8a9b58a48b86c456a0593427fbe217d6f193b404 aa8039f593df6628ef749f0c7bcb462976d45419

Have you found anything relevant by searching the web?

No.

Any other information, logs, or outputs that you want to share?

genhtml output segment for Foo.scala with Bazel 3.7.1 default Jacoco (0.8.3):

       5                 :          1 : object Foo {
       6                 :          1 :   val message = "hello world"
       7                 :          1 :   val message1 = 1
       8                 :            : 
       9                 :            :   def testLambdas(a: Int) = {
      10                 :          1 :     println(message1)
      11                 :          1 :     val foo = Foo(a, message, Set.empty[String])
      12         [ +  - ]:          1 :     foo match {
      13         [ +  + ]:          1 :       case Foo(a, _, _) if a > 10 =>
      14                 :          1 :         println("a is bigger than 10")
      15                 :            :       case _ =>
      16                 :          1 :         println("a is not bigger than 10")
      17                 :            :     }
      18                 :          1 :     println(foo)
      19                 :            :     for {
      20                 :          1 :       a <- List(foo)
      21                 :            :       b <- List(2)
      22                 :            :       c <- List(3)
      23         [ +  + ]:            :       d <- if (foo.a < 10) {
      24                 :            :         List(4)
      25                 :            :       } else {
      26                 :            :         List(14)
      27                 :            :       }
      28                 :            :     } yield a.a + b + c + d
      29                 :            :   }
      30                 :            : 
      31                 :            :   def main(args: Array[String]) {
      32                 :          1 :     testLambdas(1)
      33                 :          1 :     testLambdas(11)
      34                 :            :   }
      35                 :          1 : }

With Jacoco upgraded to 0.8.3 with Scala 2.12 lambda fix patch:

       5                 :          1 : object Foo {
       6                 :          1 :   val message = "hello world"
       7                 :          1 :   val message1 = 1
       8                 :            : 
       9                 :            :   def testLambdas(a: Int) = {
      10                 :          1 :     println(message1)
      11                 :          1 :     val foo = Foo(a, message, Set.empty[String])
      12         [ +  - ]:          1 :     foo match {
      13         [ +  + ]:          1 :       case Foo(a, _, _) if a > 10 =>
      14                 :          1 :         println("a is bigger than 10")
      15                 :            :       case _ =>
      16                 :          1 :         println("a is not bigger than 10")
      17                 :            :     }
      18                 :          1 :     println(foo)
      19                 :            :     for {
      20                 :          1 :       a <- List(foo)
      21                 :          1 :       b <- List(2)
      22                 :          1 :       c <- List(3)
      23         [ +  + ]:          1 :       d <- if (foo.a < 10) {
      24                 :          1 :         List(4)
      25                 :            :       } else {
      26                 :          1 :         List(14)
      27                 :            :       }
      28                 :          1 :     } yield a.a + b + c + d
      29                 :            :   }
      30                 :            : 
      31                 :            :   def main(args: Array[String]) {
      32                 :          1 :     testLambdas(1)
      33                 :          1 :     testLambdas(11)
      34                 :            :   }
      35                 :          1 : }

[c-mita]

I’ve done some more investigation into this and have made promising progress.

• Jacoco’s filters operate on AbstractInsnNode objects (https://asm.ow2.io/javadoc/org/objectweb/asm/tree/AbstractInsnNode.html). Our branch analysis doesn’t deal with these, only interesting itself with Jacoco’s internal Instruction objects.
• We can track these objects in the same way Jacoco’s regular analysis does as we walk the ASM tree, map them to Instruction objects, pass them to the Filter set, and record which are to be ignored.
• When we process the results to generate branch informations from the Instruction list, we skip over any who’s corresponding AbstractInsnNode is ignored.

I don’t grok what’s going on MethodProbesMapper::visitEnd, but I think skipping over Instructions that the filters have determined to be “ignored” is safe.

Note that Filters have three potential output calls to IFilterOutput:

• IFilterOutput::ignore
• IFilterOutput::merge
• IFilterOutput::replaceBranches

It isn’t clear to me yet what needs to be done to support more than just ignore. However, this would cover most cases as the others are used by very few filters.

I have a WIP branch here: https://github.com/c-mita/bazel/tree/jacoco_filter

Non-Scala/Kotlin demonstrations of the problem of extraneous branches can be demonstrated in Java with string switches:

public int foo(String myString) {
  switch (myString) {
    case "AA":
      return 1;
    case "BB":
      return 2;
    default:
      return -1;
  }
}

The “expected” number of branches here is 3, however the java compiler will emit more than double this number. Jacoco normally filters these extra branches.

[gergelyfabian]

The only “quick” solution I can think of is to omit branches on a line if Jacoco’s analyzer says there are zero branches in total, otherwise report all branches. This only works if all branches on a line have been filtered. I don’t really want to do this; it doesn’t really solve the problem and can lead to “surprising” results.

I also considered this solution in #12710 and dropped it for the same reason.

Since BranchDetailAnalyzer already uses internal Jacoco classes (e.g. ClassProvesAdapter), it might be able to apply the same filters Jacoco does (see Filters:all). Doing this would also require our own IFilterOutput implementation. These are internal classes and interfaces; going down this road could make updating the Jacoco version far more difficult if these change, making this a much more expensive approach and I don’t believe this is worth it.

I think that Bazel is already quite much dependent on Jacoco, e.g. updating to Jacoco 0.8.5 (or higher) will need some changes in Bazel itself. So if Bazel is trying to provide an alternate implementation for branch coverage, and already uses internal Jacoco classes like ClassProvesAdapter, then I think it should also provide its own IFilterOutput implementation and do filtering appropriately. I’d prefer this solution than ignoring filtering at all. If Bazel ignores Jacoco filtering, then coverage data will diverge more and more from an original Jacoco report, as Jacoco adds more and more filtering. Kotlin is one good example, but even for Java there is some filtering logic already. My personal motivation is Scala, where I’m using a fork so far to add filtering for Scala code (a lot needed).