EcsRunLauncher looking for task definition where it doesn't need to

Dagster version

0.15.5

What’s the issue?

Dagster setup with EcsRunLauncher is looking for task definition in the launched tasks where it doesn’t need it. We have dagster.yaml where the EcsRunLauncher is configured to look for task definition (arn) as configured in an environment variable, say FOO. dagster.yaml is built into the image that is used by the containers of task definitions of daemon and dagit. When they need to ask the launcher to launch the task, the launcher can find FOO in the env. where it started (because I supply it to those container environments), finds it and successfully spawns a task using arn in FOO. However that launched container (run) soon exits because something there wants to find FOO too, although it’s not useful.

I cannot normally give FOO to the container in the task definition for Launcher whose arn needs to be in FOO. That would be cyclic dependency unless i hardcode the value (not a good practice in AWS CloudFormation).

To workaround this, I also define FOO env var for the container in the task definition i reserve for the Launcher, but leave it blank. Now everything works, which indicates to me that something in the launched task is trying to merely find FOO and doesn’t actually need its value.

What did you expect to happen?

FOO in above eg. should not be needed in the container that is launched by the EcsRunLauncher. It only needs to be defined in the containers of Dagit and Daemon since that is where the Launcher starts and looks at the configuration about what to do (which task-arn to use and so on).

How to reproduce?

In dagster.yaml:

...

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator

run_launcher:
  module: dagster_aws.ecs
  class: EcsRunLauncher
  config:
    task_definition:
      env: FOO

...

Create 4 task definitions and attach a container to each of them:

1. Launcher task-def --> container named run and task-def arn = some-arn with image img1
2. Usercode task-def --> container with image img2 --> env var has FOO=some-arn
3. Dagit task-def --> container with image img3 --> env var has FOO=some-arn --> image has dagster.yaml
4. Daemon task-def --> container with image img4 --> env var has FOO=some-arn --> image has dagster.yaml

• Start Fargate services for (2), (3) and (4) above.
• Connect to Dagit and launch some job-run. This will cause the EcsRunLauncher to eventually launch a Task using task-definition some-arn it gets from env-var FOO.
• The task starts, run container soon exits and logs show that the startup code there was trying to find FOO even in that launched container, and it didn’t exist so it exited with error.

Deployment type

No response

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[johannkm]

Thanks for the report, we’ll get a fix out

[johannkm]

@ustulation that’s correct, it will only need to be set in processes that use the run launcher so Dagit and the Daemon.