`_cleanup_stale_resources()` deregisters unrelated ECS task definitions

We observe that Dask Cloud Provider deregisters ECS task definitions that it did not create. This screenshot shows task definitions that:

1. Dask Cloud Provider did not create
2. Do not get marked as INACTIVE (i.e. unregistered) when Dask Cloud Provider hasn’t been run

We haven’t fully characterized this, but it appears that the current version of _cleanup_stale_resources() is too aggressive in its de-registering of task definitions.

[capdevc]

@jacobtomlinson

This bit a user of dask-cloudprovider at my company. I don’t think the tags on task definitions are being checked in the same way that they are on clusters.

from: https://github.com/dask/dask-cloudprovider/blob/64aad0b702a9aa309d0de22b568a58e0199e711c/dask_cloudprovider/providers/aws/ecs.py#L1258-L1274

It looks like any task without a tag for “cluster” (line 1265) matches line 1269 and gets deregistered. If all that’s supposed to happen is checking for tasks tagged with a cluster not in the active list, then it’s a pretty easy local fix right in that code block to flip the None check and or. I assume it’s supposed to be checking for the DEAFULT_TAGS as well?

[AndrewHannigan]

@jacobtomlinson @martindurant I have an open PR for this when you have a chance to review.