2022 Summer of Docs Proposal

Description

From email discussions on the Apache listserv, I’d like to propose the following Google Summer of Docs program for this wonderful project. These are just a first take and each of the sections should warrant its own deep-dive discussion (maybe on a GitHub issue or discussion). I think each section can benefit from interactive visuals which we can include as part of the summer of docs.

Airflow with Your Team

Working together on complex software engineering is never an easy task. These docs contain conventions and suggestions on how you can effectively work together on your team.

Git-Ops

We recommend that teams work together on an Airflow project with git. Teams should yearn to have one and only one main branch which reflects what is deployed to their production environment. Every change into the main branch should happen via a pull request so it’s tested and reviewed.

Continuous Integration

We recommend automatically enforcing as many rules as possible by using tools like GitHub’s Super Linter. For Python projects, this means linting your code with Black, isort and Flake8 (the same linters required to publish code here).

We also recommend running as many tests as possible to test the arity of your methods and the accuracy of your SQL scripts.

Staging Environment

For many teams, a true staging environment is not feasible because data constantly changes and data warehouses are very large. I’m really not sure what people’s takes are here, I think a staging environment isn’t worth the effort and instead running DAGs with the built-in feature flagging is the best way to maintain velocity.

Hotfixes

Instead of a staging environment, your team may consider a hotfix environment which has parity with production and additionally contains hotfix changes that need to be deployed immediately. We recommend deploying the hot fix server from pull requests.

Use case/motivation

I do DevOps for a data engineering team heavily reliant on Airflow to manage our scripting. We have iterated several times on how we work together with Airflow and would like to collaborate with other teams on how they best use Airflow to improve our process.

I think the issues proposed above are common problems many teams face, and having helpful documentation will point us all towards how to use Airflow amongst teams.

Related issues

None that I know of that specifically call out working as a team.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project’s Code of Conduct

[shicholas]

would love to, will email your address listed on your blog, thanks!

[094459]

I am happy to pitch in and collaborate on this with you. I have some experience/exposure to this space, maybe we can have a quick call to sync up and share ideas?