Ember Server - Delayed Processing with TLS Requests

I switched my kinesis-mock to use the Ember backend per a discussion w @djspiewak and @ChristopherDavenport. After doing so, one of my functional tests started failing non-deterministically: https://github.com/etspaceman/kinesis-mock/blob/main/src/fun/scala/kinesis/mock/DescribeStreamSummaryTests.scala#L42-L56

This will run 11 requests in parallel and check the response. In successful iterations, this test takes about 1.5 seconds in total (including the test setup). In the failed scenarios, it times out after 30 seconds.

I did some research into what is happening. Attached the logs from one of my CI runs. If you do a search for “2021-06-20 15:50:14”, you’ll find the logs from the server when the test starts. 10 of the 11 streams get processed at this timestamp. stream7, however, isn’t processed until “2021-06-20 15:50:45”. So almost exactly 30 seconds later.

There is a ShutdownTimeout in Ember that is set to 30 seconds, which might be a clue. I’m kind of grasping at straws here though.

I have also only seen this on TLS enabled servers. Might be a coincidence. I’ll mention if I see it on a plain-text server.

6_Print Docker Logs.txt

[etspaceman]

I believe this was resolved with a recent update from FS2. Closing.

[etspaceman]

That is a good callout @djspiewak - I’m using ExecutionContext.global in Blaze but not in Ember (which would use the fixedThreadPool from IOApp). That would explain a couple of things:

• Why its only showing up in the Ember server
• Why I cant recreate it locally

I’m seeing if I can invoke the issue in the Blaze server here: https://github.com/etspaceman/kinesis-mock/pull/124