Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Ability to better support odd scheduling time

See original GitHub issue

Description

So I have been encountering odd time schedule request recently, say some updates need to be done on every second and third Friday each month, 4pm.

Tried to search for solution but could not find a perfect ones, I can think of the following solutions:

use a cron and a separate DAG
embed trigger in the actual task logic, that is to check whether the code should be executed in that date, otherwise return success (equivalent to skip and mark success)

Drawbacks (at least the one i can think of):

it creates extra DAG, and when you have a bunch of these requests, it’s not really clean. Besides, you need to add equal number of ExternalTaskSensor to preserve inter-DAG dependencies. It also however, will need to provision other dependent resources (such as Kafka consumers)
not very generic and reusable apparently, and if you are going for a full containerized solution - each Task is executed in container - you also need to start a container first to reach that simple if-else check, which seems a bit wasteful

Use case / motivation

Basically I have a bunch of extraction scripts that need to be executed at very different and very weird times, very straight forward (?), without the need to replicating provision tasks, too many inter-DAG dependencies

Potential solutions I have in mind:

not touching Airflow core (workaround)

inherit and extend the Operators i am using - troublesome and needs to repeat for multiple operators

touching Airflow core

an extra tuple params allowed_dates passed into the operators (BaseOperator), and then inject the date check right before the operator execution, essentially giving all operator the ability to be programmatically skipped in any DAG - might be an overkill