Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Prefect Docker Agent Continuously Launches Containers when using Concurrency Tags

See original GitHub issue

Description

Hi everyone, I have been testing with concurrency tags and had a question about some behavior I am seeing with docker. It seems that after a task has been waiting (via concurrency tag limit) for 10 minutes, the prefect docker agent starts continuously spinning up new containers for that flow that is waiting to start. If I do a “docker ps” I could see 20 or more containers running for that flow.

Expected Behavior

While a flow is paused due to a concurrency tag limit, new docker containers are not continuously spun up by the agent.

Reproduction

It should be fairly simple to reproduce.

Create two flows that have tasks that take over 10 minutes to run.
Assign those long tasks to use the same concurrency tag.
Spin up one prefect docker agent to run the flows.
After 10 minutes check docker with “docker ps”.

Environment

Prefect Version: 0.10.6 Docker Engine Version: 19.03.6-ce Storage Type: Docker OS: Replicated the issue on both Mac and Amazon Linux 2

Optionally run prefect diagnostics from the command line and paste the information here. -->

Issue Analytics

State:
Created 3 years ago
Comments:16 (8 by maintainers)

Top GitHub Comments

1reaction

cvroomancommented, Jul 21, 2020

@cicdw sorry for the delay in my response - I appreciate your help and updates on this. This is great news, thanks for tracking this down! We ended coming up with an alternative flow that ran more linear (without concurrency tags) just for the time being. I’ll keep an eye out for 0.12.5 today and give it a shot!

1reaction

cicdwcommented, Jul 20, 2020

Final update @cvrooman - the required changes on the backend have been released! If you run your work off of master (or 0.12.5 after it’s released tomorrow) you shouldn’t experience the repeated redundant submissions of work

Top Results From Across the Web

Docker Agent - Prefect Docs

The Docker agent executes flow runs in individual Docker containers. This provides more isolation and control than the Local Agent, while still working...

Changelog - Prefect Docs

Fix binding of named volumes to flow containers with Docker agent - # ... Remove deprecated prefect agent local start flag --storage-labels ......

Running flows with Docker - Coordinating the world's dataflows

Learn how to build Prefect deployments that create flow runs in Docker containers.

Infrastructure - Prefect 2 - Coordinating the world's dataflows

Environment variables to set for the container. image, An optional string specifying the tag of a Docker image to use. Defaults to the...

Docker Agent - Prefect Docs

Agent which deploys flow runs locally as Docker containers. Information on using the Docker Agent can be found at ...