Cannot write logs to volume not supporting unix-style permissions

Apache Airflow version: v2.0.0b2

Kubernetes version (if you are using kubernetes) (use kubectl version): v1.16.13

Environment:

• Cloud provider or hardware configuration: Azure AKS
• OS (e.g. from /etc/os-release): official airflow image

What happened:

When configuring the worker to write logs to a PersistantVolumeClaim in Kubernetes that does not support unix-style permissions (AzureFiles in my case), writing the logs - and hence the whole job - fails.

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 50, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 86, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 179, in task_run
    ti.init_run_context(raw=args.raw)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1922, in init_run_context
    self._set_context(self)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 54, in _set_context
    set_context(self.log, context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 173, in set_context
    handler.set_context(value)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 54, in set_context
    local_loc = self._init_file(ti)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 256, in _init_file
    os.chmod(full_path, 0o666)
PermissionError: [Errno 1] Operation not permitted: '/opt/airflow/logs/dagname/jobname/2020-11-27T20:57:00+00:00/1.log'

What you expected to happen:

Airflow should recognize that permissions are not available on the volume or allow disabling setting permissions for logfiles via config.

How to reproduce it:

Try setting the default log path to a volume not supporting permissions.

[cris-b]

I have hit this with azure files in 1.10. The way I worked around it was to set the uid/gid perms on the pvc, but this effectively forces all worker images, the scheduler and the webserver to have a hard requirement on running as the same UID which may not be possible.

[potiuk]

Not strictly 2.0 related (same error in 1.10 I guess) but might be an easy one to fix.