ECSOperator returns last logs when ECS task fails

Description

Currently when the ECSOperator fails because the ECS task is not in ‘success’ state it returns a generic message like that in Airflow alerts that doesn’t have much value when we want to debug things quickly.

This task is not in success state {<huge JSON from AWS containing all the ECS task details>}

Use case / motivation

This is to make it faster for people to fix an issue when a task running ECSOperator fails.

Proposal

The idea would be to return instead the last lines of logs from Cloudwatch (that are printed above in Airflow logs) so when we receive the alert we know what failed in the ECS task instead of having to go to Airflow logs to find it. This feature would involve changes there I think:

• https://github.com/apache/airflow/blob/2ce6e8de53adc98dd3ae80fa7c74b38eb352bc3a/airflow/providers/amazon/aws/operators/ecs.py#L354
• https://github.com/apache/airflow/blob/2ce6e8de53adc98dd3ae80fa7c74b38eb352bc3a/airflow/providers/amazon/aws/operators/ecs.py#L375

[uranusjr]

We can probably use collections.deque for better performance, but aside from that, sounds good to me! Would you be interested in opening a pull request for this?

p.s. I don’t think IndexError can ever be raised?

[potiuk]

Assigned you @pmalafosse