KubernetesPodOperator does not have any logs

Apache Airflow version: 2.0.2

Kubernetes version (if you are using kubernetes) (use kubectl version):

...
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): linux
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

With DEBUG logging on, when using a simple hello world KubernetesPodOperator no logs are produced in the executor either when there is a problem with the setup or when everything is working.

What you expected to happen: Executor has logs showing the error or successful invocation.

There are no logs at all. If I force it to be interactive by changing https://github.com/apache/airflow/blob/476d0f6e3d2059f56532cda36cdc51aa86bafb37/airflow/cli/commands/task_command.py#L236 (comment out the with clause and just invoke _run_task_by_selected_method as is done for interactive) I get logs in the executor pod. Whatever _capture_task_logs is trying to do it seems to hinder rather than help

How to reproduce it: With helm chart version 8.1.1 and airflow 2.0.2 built from source, and logging config:

[logging]
    colored_console_log = False
    logging_level = DEBUG
    remote_logging = False
    donot_modify_handlers = True

run a DAG that uses the from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator operator. Note that there is no logging in the executor. Modify task_command.py to call _run_task_by_selected_method without try _capture_task_logs and see that the operator has a lot of useful logging including the pod spec (which is pretty much essential to diagnose any issues).

Anything else we need to know:

[vbarbaresi]

We are affected by this issue as well slaupster’s hack to remove _capture_task_logs() worked around the problem. It makes sense for us to use stdout/stderr for all tasks logs instead of writing them in a bucket: we collect all our infrastructure logs this way

[github-actions[bot]]

This issue has been closed because it has not received response from the issue author.