Windows support for Airflow

Description

Currently, the airflow project uses PEP-3143 style daemons to launch tasks (as implemented in https://pypi.org/project/python-daemon/), however this is targeted towards unix daemons. As a result, running airflow on windows requires multiple levels of abstraction each with their own problems. Would it be possible to use something like daemoniker (https://daemoniker.readthedocs.io/en/latest/) to launch tasks? What are the challenges and issues?

In machine learning workflows, with large datasets, it is a huge time-saver if the pipeline tasks can be run on the GPU. WSL 1 does not support GPU passthrough, docker through WSL 2 supports GPU passthrough only with the Insiders build, additionally it has issues with networking when connected to VPN (https://github.com/microsoft/WSL/issues/5068).

Use case / motivation

Natively running airflow without WSL 1/2 or docker on Windows. This is helpful in cases where the company ecosystem is windows-based.

Possible implementation

The daemon module is only used to daemonize the scheduler and webserver. Here’s a sample code that runs the scheduler (airflow origin/v1-10-stable) using daemoniker, comments are welcome:

# airflow/bin/cli.py
from daemoniker import Daemonizer

...

if args.daemon:
    with Daemonizer() as (is_setup, daemonizer):
        if is_setup:
            pid, stdout, stderr, log_file = setup_locations("scheduler",
                                                    args.pid,
                                                    args.stdout,
                                                    args.stderr,
                                                    args.log_file)
        
        _is_parent = daemonizer(
            pid,
            stdout_goto=stdout,
            stderr_goto=stderr
        )

    job.run()

[potiuk]

I think it would be great if someone could invest in Windows support. I believe there are few things - not only the daemon model but also Local Executor uses fork mechanisms which won’t be able on Windows, also there might be some problem if you want to use Celery Executor on Windows: https://www.distributedpython.com/2018/08/21/celery-4-windows/ There are few POSIX-compliant packages used as well with might not work on Windows. And automated testing might be a problem since we are using Docker. It looks like quite a big effort to invest…

[casra-developers]

We have a go, I will create a fork and CC you @potiuk in the PR. There is probably a lot of things we need to do since the only goal was to implement enough functionality for Dask to run properly.