faq.rst provides incorrect instructions for reducing scheduling latency

Regarding the FAQ note “How to reduce airflow dag scheduling latency in production” - Per https://github.com/apache/airflow/blame/f097ae39a7243bd25d4d26664bc259981b2ba217/docs/faq.rst#L209:

User should consider to increase scheduler_heartbeat_sec config to a higher value (e.g 60 secs) which controls how frequent the airflow scheduler gets the heartbeat and updates the job’s entry in database.

However, since scheduler_heartbeat_sec is used as a duration (not as heartbeats/sec rate) in airflow/jobs/scheduler_job.py, increasing it to 60 (from the default 5 sec) would actually cause scheduling to become more sluggish, thus increasing latency.

[ashb]

Yes. The task lag was fixed by AIP-15 and in my tests the delay between tasks is down to 0.18s

Try out 2.0.0beta2?

[vitaly-krugl]

Hi @ashb - The reason I was looking at scheduler_heartbeat_sec is that I am trying to improve the performance of system-level tests in my airflow-based app.

What I am seeing is that even under very low utilization, Airflow adds latency of 4-5 seconds for executing each task. I was looking at airflow.cfg options to tune to eliminate this 4-5 second per task latency on my testing setup. I haven’t been able to find any combination of options that would reduce the latency below 4-5 seconds.

Any suggestions about how to eliminate these Airflow latencies for my testbed?

My test setup: one DAG with two tasks: Task A and Task B, with A >> B relationship. Implementation is in python. Each one of the python callbacks does minimum (almost no-op) work that shows up in the logs at < 1 sec. However, each dagrun takes upwards of 9 seconds and I observe from the timestamps (dagrun, Task A start/end and Task B start/end) that there are 3-5 second gaps between dagrun and Task A start, as well as between Task A end and Task B start.