S3 remote logging not working for airflow server components

Apache Airflow version: 2.0.1

Environment:

• Cloud provider or hardware configuration: on my laptop
• OS (e.g. from /etc/os-release): MacOS Majave 10.14.6
• Kernel (e.g. uname -a): Darwin Wongs-MBP 18.7.0 Darwin Kernel Version 18.7.0: Tue Jan 12 22:04:47 PST 2021; root:xnu-4903.278.56~1/RELEASE_X86_64 x86_64

What happened:

configured remote logging to S3 bucket, only the logs of DAG runs appeared in the bucket. logs of airflow server components: scheduler, web server, etc did not appear

What you expected to happen:

all logs go to S3 bucket

How to reproduce it:

1. follow the quick start guide in https://airflow.apache.org/docs/apache-airflow/stable/start/local.html

2. before starting web server set the following variables:

export AIRFLOW__LOGGING__REMOTE_LOGGING=True
export AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=s3://my-bucket/
export AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID=my_remote_logging_conn_id

1. start the web server and set your S3 connection settings in the web server “connections” section.

Conn Id * my_remote_logging_conn_id
Conn Type  S3
Extra {"region_name": "nyc3",
 "host": "https://nyc3.digitaloceanspaces.com",
 "aws_access_key_id": "xxx",
 "aws_secret_access_key": "xxx"}

1. Restart the web server
2. Start the scheduler in another console window (setting the same env variables)
3. Execute a DAG
4. Head to your S3 bucket UI, you will see only logs of DAG runs appear.

[mik-laj]

Airflow handles some logs in a special way and unfortunately, it is not easy to unify this as our logs have different characteristics.

Logs for tasks are saved to files because they are small and we can upload them after completing the tasks. When they are sent to object storage it is much easier to read to them, but each time you add a new line, we need to get all the contents and upload a new file with the full log contents. Object storages have limited support for appending operations. On the other hand, logs for other components are an endless stream of data that cannot be stopped to send the content. For this reason, conventional tools fit better because they are optimized to handle such operations.

I don’t think we can fix this problem, but we can update the documentation to better describe these project assumption.

Related issue: https://github.com/apache/airflow/issues/10593

[kakarukeys]

Since this is not classified as a bug, I will close this. Thanks for your help.