FireCloud - cromwell losing jobs?

We are seeing infrequent situations where JES says a call is done but cromwell thinks it is still running.

I see the call starting in the logs:

2017-02-08 18:55:58,500 cromwell-system-akka.dispatchers.engine-dispatcher-963 INFO - WorkflowExecutionActor-fa7e25a2-f51f-4763-9f8a-5a2e5cd1c954 [UUID(fa7e25a2)]: Starting calls: pon_gatk_workflow.PadTargets:NA:1

Then the only suspicious things I see later in the logs are these messages (which could be completely unrelated however they do start to appear 2.5 minutes after the job completes on JES):

2017-02-08 19:17:07,588 cromwell-system-akka.dispatchers.backend-dispatcher-424 INFO - The JES polling actor Actor[akka://cromwell-system/user/cromwell-service/$b/$a/$Enn#762671444] unexpectedly terminated while conducting 100 polls. Making a new one...
java.lang.NullPointerException: null
2017-02-08 19:17:07,588 cromwell-system-akka.dispatchers.backend-dispatcher-246 ERROR - null

For future reference by a FireCloud admin the operations id is ELSl1PihKxjdhp-6gvr3weYBILma7PWMHyoPcHJvZHVjdGlvblF1ZXVl and the workflow id is fa7e25a2-f51f-4763-9f8a-5a2e5cd1c954 and the workflow was aborted 3PM Feb 8.

[kcibul]

Given the potential impact of this, I’d like to prioritize this @geoffjentry @katevoss . I think the the new policy is to add the priority label, right? and bug wise, if confirmed, I’d say this is a P2 as there is a painful workaround (but still a priority ticket).

[ruchim]

Spoke to @dvoet and he mentioned that since FC hasn’t seen any more of the same symptoms (NPE’s and jobs getting lost) for some time now, we can consider this issue resolved for now.