question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Workflows with action policies can deadlock

See original GitHub issue

Given the attached pack (useful for meditation), the system deadlocks when the action _dispatcher_pool (50 entries) gets filled with workflows, and all workflow actions are subsequently held in the _work_buffer queue indefinitely.

One possible solution would be to limit the number of workflows in the _dispatcher_pool to one less than half the total entries, in order to leave room for action execution.

deadlock.tar.gz

Deadlock pack

This pack contains actions for testing policy deadlock and other behaviors

Actions

  • delay2 - a two second delay
  • delay20 - a twenty second delay
  • delay2test - a workflow with delay2 followed by delay20

Policies

  • delay20 - limit concurrency to 1
  • delay2test - limit concurrency to 10

Testing

Launch multiple instances of the workflow:

  • for i in seq 1 100; do st2 action execute deadlock.delay2test; done

Without the delay2test policy, the actions deadlock

Issue Analytics

  • State:closed
  • Created 7 years ago
  • Comments:22 (21 by maintainers)

github_iconTop GitHub Comments

1reaction
Kamicommented, Jul 18, 2016

@bri365 Thanks for the feedback.

I will also wait on @lakshmi-kannan and others to chime in and if they agree with my proposed approach, I will go ahead and implement it 😃

0reactions
Kamicommented, Jul 27, 2016

Awesome, glad to hear it.

And yes, they should be executed in order as requested. Is the ordering issue the same as one described in #2797?

If so, let’s continue there and if not, it’s probably the best to open a new issue and we will have a look 😃

Read more comments on GitHub >

github_iconTop Results From Across the Web

custom object - Workflow , Trigger dead lock scenario
Custom validation rules and duplicate rules are not run again. Executes processes. If there are workflow flow triggers, executes the flows. The ...
Read more >
Tuning Action Runner Dispatcher Pool Size
In some situations, such as if you have a lot of action-chain workflows with a delay policy, this limit can cause a deadlock...
Read more >
35 Customizing Workflow - WebCenter Sites
Deadlock actions, which are executed when an asset needs a unanimous vote in order to move to the next state, but the voters...
Read more >
Workflow graph with deadlock and lack of synchronization ...
Figure 4 shows a workflow graph without any tasks as it occurs in the middle part of the process in Fig. 2 and...
Read more >
Github actions workflow on self hosted runners failing due to ...
It's for reasons like this the guidance is to not run multiple runner agents on the same machine. Resource contention and deadlocks will...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found