Exceptions thrown in a Flink streamlet are not logged anywhere.

Describe the bug A clear and concise description of what the bug is. Exceptions thrown in a Flink streamlet are not logged anywhere. for instance throwing an exception in buildExecutionGraph or throwing an exception in a

readStream(in).map {
  // throw here
}

Is not showing up in any logs.

To Reproduce Modify for instance taxi-ride app to throw exceptions in the processor.

Expected behavior To see exception stacktraces in the logs, and a restart of the pod.

[Lockdain]

Hi @franciscolopezsancho

I apologize in advance in case I’ve missed some parts of your investigation.

The interesting question is to investigate how Cloudflow together with Lyft operator under the hood handles automatic job recovery in cases when the job fails due to some “recoverable” reasons. There is a great example available on YouTube - here.

[Lockdain]

Hi @franciscolopezsancho

Thanks for the great investigation.

We’ve also observed a faulty behavior in case of JM pod restarts. So are you able to walk through the following steps using your example?

1. Run a Flink job which is totally correct and doesn’t contain any hardcoded exceptions.
2. Make a few curl’s to the entrypoint in order to pass some records through the job.
3. Force delete the job manager pod (a second case - try to shutdown gracefully).
4. Wait until K8s will schedule and run a new JM pod.
5. Look at the logs (new JM, old TMs)

In the default (not HA-mode) JM becomes a single point of failure, so the Flink operator relies on the default K8s politics considering pod re-scheduling. So it would be also great to repeat the same scenario against some Flink cluster with JM running in HA-mode.

Thanks in advance for any assistance!