Surprisingly slow on GraalVM vs OpenJDK

I have the least-minimized bug of all time for you. I’m going to try to minimize it at some point, but I wanted to file it here just to get it on your radar somewhere so you know it exists.

For some reason, some ScalaJS code takes an incredibly long time to run when the system JVM is GraalVM (20.1.0 CE, target 1.8, to be precise). To be very clear, I’m not running the javascript on GraalVM, I’m just running sbt. To be even more clear, it’s the running of the generated javascript which is slow, not the compilation. No, it doesn’t make any sense.

https://github.com/djspiewak/cats-effect/tree/bug/scalajs-weirdness

There’s a tag where the problem is exhibited. Run sbt 'testOnly *PureConcSpec'. When running under GraalVM, it takes about 90-120 seconds to complete. When running under OpenJDK, it’s closer to 15 seconds. This is part of why it’s so weird: the disparity is massive. Also this happens in the ubuntu-latest runner in GitHub Actions as well, so it isn’t just my laptop.

~/.../Scala/cats-effect (ce3) λ java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM GraalVM CE 20.1.0 (build 25.252-b09-jvmci-20.1-b02, mixed mode)
~/.../Scala/cats-effect (ce3) λ node --version
v14.5.0

Oh, and if that wasn’t weird enough, ClockSpec in that same project also exhibits the same disparity, but IOSpec does not. So it’s like… very much not consistent.

I apologize for how completely unminimized this is.

[sjrd]

This is a known issue and 100% Graal’s fault. When a JVM running on Graal spawns a process named node, it chooses GraalJS’ version of node over the system one. And GraalJS is terrible performance wise on startup.

There is no valid reason for Graal to impose its version of node. It shouldn’t do that. Please report it to them that they should stop doing that because it’s terrible.

[djspiewak]

@sjrd @shelajev I ran console.log(process.execPath) within the following contexts:

• node -e from a fresh shell
• Ammonite "node -e ...".!! from the same shell
• Embedded within a scala.scalajs.js.eval statement in one of my tests which doesn’t appear massively slower on Graal
• Embedded within a scala.scalajs.js.eval statement in one of my tests which does appear slower

java -version results on each:

openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM GraalVM CE 20.1.0 (build 25.252-b09-jvmci-20.1-b02, mixed mode)

The results of the process on each were identical: ~/.nvm/versions/node/v10.5.0/bin/node This almost definitively proves that the initial conclusion was in err: Graal isn’t redirecting anything. I’ll be noting that on my tweet thread.

This does still leave open the mystery of why we see such radical performance differences, but I have a theory there… First off, it’s very possible that Graal somehow ends up ahead of NVM on the GitHub Actions runner path. That’s testable, though I don’t have time to go through those shenanigans right now. If that’s the case, then Graal would indeed be intercepting node on GitHub Actions runs, but only due to the fact that its version of node (and npm) are first on the PATH. I still would prefer they didn’t ship a node, but… this isn’t really dodgy, just not what I would do.

Regarding my laptop… I went back and tried to reproduce the performance difference between VMs and wasn’t able to. I do all my development on Graal and the performance was still bad, but as it turns out, node run from OpenJDK is just as bad now. It’s very possible that I changed something weird in the implementation which caused the problem locally.

At any rate, at this point, I’m relatively certain that Sebastian’s initial understanding (“When a JVM running on Graal spawns a process named node, it chooses GraalJS’ version of node over the system one.”) is in err. I think it’s still worth digging into the GitHub Actions situation a little more closely, and I’ll do that, but at the very least I’m not able to reproduce that symptom on my laptop.

I apologize for my misunderstanding.