Example: How to create a multi-step job

Add an example on how to create a multi-step job.

Two possible solutions:

• Maintain state in task_data, update and reschedule
• ~Schedule the next step at the very end of the current step using the SchedulerClient (not guaranteed atomic)~
• Schedule next step in the CompletionHandler

[kagkarlsson]

I may have been a bit quick to answer here:

Could you expand on why the second option (scheduling the next task at the end of the step) is not atomic? Doesn’t saving the newly scheduled task happen in the same transaction as completing/deleting the current one?

Answer:

The CompletionHandler is currently not transactional as it up until now only have had a single jdbc-statement. Might be needed to better support multistep-jobs.

For the second option (in the description), the next step is actually scheduled at the very end of the ExecutionHandler. The CompletionHAndler will run after the full ExecutionHandler has run. If we put it at the very end of the ExecutionHandler there may be some edge-cases. The one I can think of is:

• Execution A runs. Schedules execution B at the very end of its ExecutionHandler. Complete of execution A fail due to db/network-problems. Execution B remains scheduled and runs. Execution A is discovered as “dead” since it completion failed and the lock remained. Execution A is “revived” (depends on DeadExecutionHandler, customizable) and runs again after execution B have already run, and may actually schedule a new Execution B. To prevent you could for example make sure that a second Execution B will have no effect (make idempotent or similar)

[kagkarlsson]

Merged a PR enabling scheduling the next step in the CompletionHandler, safely. See JobChainingUsingSeparateTasksMain.java

I have also fixed so that immediateExecution works for scheduling that happens in the CompletionHandler, so enableImmediateExecution() will cause anything scheduled to now() (or before) to run directly, bypassing the pollingInterval.

I think what remains for these use-cases is reliably scheduling two or more new executions in a CompletionHandler. It is probably a niche use-case, but would be nice to support. Requires transactional CompletionHandlers.

(The changes mentioned are not yet release, but will be soon)