Workflows with action policies can deadlock
See original GitHub issueGiven the attached pack (useful for meditation), the system deadlocks when the action _dispatcher_pool (50 entries) gets filled with workflows, and all workflow actions are subsequently held in the _work_buffer queue indefinitely.
One possible solution would be to limit the number of workflows in the _dispatcher_pool to one less than half the total entries, in order to leave room for action execution.
Deadlock pack
This pack contains actions for testing policy deadlock and other behaviors
Actions
delay2
- a two second delaydelay20
- a twenty second delaydelay2test
- a workflow with delay2 followed by delay20
Policies
delay20
- limit concurrency to 1delay2test
- limit concurrency to 10
Testing
Launch multiple instances of the workflow:
for i in
seq 1 100; do st2 action execute deadlock.delay2test; done
Without the delay2test
policy, the actions deadlock
Issue Analytics
- State:
- Created 7 years ago
- Comments:22 (21 by maintainers)
Top Results From Across the Web
custom object - Workflow , Trigger dead lock scenario
Custom validation rules and duplicate rules are not run again. Executes processes. If there are workflow flow triggers, executes the flows. The ...
Read more >Tuning Action Runner Dispatcher Pool Size
In some situations, such as if you have a lot of action-chain workflows with a delay policy, this limit can cause a deadlock...
Read more >35 Customizing Workflow - WebCenter Sites
Deadlock actions, which are executed when an asset needs a unanimous vote in order to move to the next state, but the voters...
Read more >Workflow graph with deadlock and lack of synchronization ...
Figure 4 shows a workflow graph without any tasks as it occurs in the middle part of the process in Fig. 2 and...
Read more >Github actions workflow on self hosted runners failing due to ...
It's for reasons like this the guidance is to not run multiple runner agents on the same machine. Resource contention and deadlocks will...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
@bri365 Thanks for the feedback.
I will also wait on @lakshmi-kannan and others to chime in and if they agree with my proposed approach, I will go ahead and implement it 😃
Awesome, glad to hear it.
And yes, they should be executed in order as requested. Is the ordering issue the same as one described in #2797?
If so, let’s continue there and if not, it’s probably the best to open a new issue and we will have a look 😃