How will Activity with different parameters work?

Problem Statement

Consider a video conversion activity, steps,

1. Activity1: Download blob to local file filePath
2. Activity2: run ffmpeg on filePath
3. Activity3: upload file filePath + “.mp4”

In this case filePath cannot be unique to execution, because what if temp file clean up process kicks in and deletes the file as process was recovered from crash.

So in first execution lets say filePath was c:\file\1.dat

Execution is interrupted and before firing Activity1 we change filePath to c:\file\2.dat because file 1.dat may or may not have been download successfully or file may not be present.

What will happen if we fire activity 1 with c:\file\2.dat argument?

1. Activity1 will be played for c:\file\2.dat or not? I assume it should be played.
2. What will happen to previous Activity1 with c:\file\1.dat? Will it cause any issue for Orchestration to finish or not? Will Orchestration hang forever as it could not finish first activity?

[ConnorMcMahon]

@ackava, I am a bit confused about your claim that Durable Tasks are not suited for large workflows that span days to months. The issues we brought up above were specifically with the non-determinism and the fact that most storage providers do not guarantee the locality of activities running on the same VM.

The Eternity framework you built looks interesting, and has a lot of the syntactic sugar that we have built on top of Durable Task Framework in Durable Functions, though with some very different design decisions around determinism and locality. I do think your decision to move mostly to blobs and tables is interesting, and I think may bypass some of the corner cases we have had to tackle over the years.

EDIT: I will say that you may want to consider some edge cases of your decision to run all activities on the same VM, such as what happens when that worker goes down mid-workflow?

[davidmrdavid]

Hi @ackava,

Sorry for causing confusion, I did delete my original answer because I wrote it thinking that you were asking about Durable Functions instead of the Durable Task Framework. I wanted to re-post it with an updated answer, but the thread progressed further before I managed to post it. In any case, let’s not worry about it, Chris and Connor have already made the main points I wanted to convey 😃 .

To answer your latest question, you’re correct, that would crash with a NonDeterministicOrchestrationException. That being said, it’s probably a bit dangerous to rely on that behavior. Ideally, there would be no way your orchestration could change a parameter value on a replay, as that goes against the coding constraints (determinism) that the framework expects of orchestrator code.

As others have pointed out, if changing this file name is unavoidable, it might make sense to combine all affected activities into just one, and to make that pipeline occur with just one activity call. That way, you encapsulate non-determinism in activities, and keep orchestrators replay-safe.