Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Upstream failed whereas Upstream is alright

See original GitHub issue

Apache Airflow version: 1.10.3 on Google Cloud Composer (so 1.10.3-composer)

Kubernetes version (if you are using kubernetes) (use kubectl version): (Managed by Google Cloud Composer)

Environment:

Cloud provider or hardware configuration: Google Cloud Composer
OS (e.g. from /etc/os-release): (Managed by Google Cloud Composer)
Kernel (e.g. uname -a): (Managed by Google Cloud Composer)
Install tools: N/A
Others: Managed by Google Cloud Composer

What happened:

Sometimes, some steps do not run because of “Failed Upstream” (orange color) whereas upstream is alright (green color).

Examples:

Additional informations:

All steps except wait_for_dump, recort_start, join and record_end are Kubernetes Pod Operator.
Re running by hand “orange” steps always solve the issue.

What you expected to happen:

If all upstream tasks of a given task are green, the given task should run, and not be in state “Failed Uptream”.

How to reproduce it: Run some DAGs (no need to be a specific DAG) on Airflow (via Google Cloud Composer), and the issue will happen randomly.

Anything else we need to know: I cannot reproduce this bug on my local version on Airflow, on MacOS (after about 10 trials) using the Kubernetes single-node cluster available in Docker for mac.

I’m not 100% sure this issue come from Airflow itself, but may also come from Google Cloud Composer.