Create an `airflow upgrade-check` command in 1.10 to ease upgrade path to 2.0.

To make it easier for users to upgrade from 1.10 to 2.0 (when it eventually comes out) we should create a single upgrade-check command in 1.10 that checks the following things. We could also have a mode that makes some of these changes in place (with confirmation from user) to automate it.

Rules

Major breaking changes:

• ConnTypeIsNotNullableRule - Not-nullable conn_type column in connection table (done)
• UniqueConnIdRule - Unique conn_id in connection table https://github.com/apache/airflow/pull/11222
• CutomOperatorUsesMetaclassRule - BaseOperator uses metaclass https://github.com/apache/airflow/issues/11038
• UsingSQLFromBaseHookRule - Remove SQL support in base_hook https://github.com/apache/airflow/issues/11039
• ChainBetwenDAGAndOperatorNotAllowedRule - Assigning task to a DAG using bitwise shift (bit-shift) operators are no longer supported
• AirflowMacroPluginRemovedRule - Removal of airflow.AirflowMacroPlugin class
• NoAdditionalArgsInOperatorsRule - Additional arguments passed to BaseOperator cause an exception https://github.com/apache/airflow/issues/11042
• MesosExecutorRemovedRule - Removal of Mesos Executor

Config related changes:

• HostnameCallableRule - Unify hostname_callable option in core section https://github.com/apache/airflow/issues/11044
• StatNameHandlerNotSupportedRule - Drop plugin support for stat_name_handler
• LoggingConfigurationRule - Logging configuration has been moved to new section
• NoGCPServiceAccountKeyInConfigRule - Remove gcp_service_account_keys option in airflow.cfg file
• FernetEnabledRule - Fernet is enabled by default
• KubernetesWorkerAnnotationsRule - Changes to propagating Kubernetes worker annotations
• LegacyUIDeprecatedRule - Deprecate legacy UI in favor of FAB RBAC UI
• TaskHandlersMovedRule - GCSTaskHandler has been moved, WasbTaskHandler has been moved, StackdriverTaskHandler has been moved , S3TaskHandler has been moved, ElasticsearchTaskHandler has been moved, CloudwatchTaskHandler has been moved
• SendGridMovedRule - SendGrid emailer has been moved
• CustomExecutorsRequireFullPathRule - Custom executors is loaded using full import path

Import changes:

• ImportChangesRule - uses a map old_operator_name -> list of possible problems so we can create a single DagBag and scan all used operators and raise information about changes. It should also suggest what providers packages users should use.

How to guide

To implement a new rule we had to create a class that inherits from airflow.upgrade.rules.base_rule.BaseRule. It will be auto-registered and used by airflow upgrade-check command. The custom rule class has to have title, description properties and should implement check method which returns a list of error messages in case of incompatibility.

For example: https://github.com/apache/airflow/blob/ea36166961ca35fc51ddc262ec245590c3e236fb/airflow/upgrade/rules/conn_type_is_not_nullable.py#L25-L42

Remeber to open the PR against v1-10-test branch.

[potiuk]

Sounds great to start with. I am sure we will have more rules added by the users, but those seem like great to create issues for. And yeah. One rule to rule them all (“naming changes”) sound great. But I’d name it “ImportChangesRule”. We renamed Hooks, and Secret Backends and Sensors - not only operators 😃.

Elder Jarek.

[kaxil]

Yup I am happy with the suggested list