Migrate Amazon example DAGs to new design

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all example DAGs related to Amazon provider. It is created to track progress of their migration.

List of paths to example DAGs:

• airflow/providers/amazon/aws/example_dags/example_s3_bucket_tagging.py
• airflow/providers/amazon/aws/example_dags/example_emr_eks.py
• airflow/providers/amazon/aws/example_dags/example_athena.py
• airflow/providers/amazon/aws/example_dags/example_sqs.py
• airflow/providers/amazon/aws/example_dags/example_eks_with_nodegroups.py
• airflow/providers/amazon/aws/example_dags/example_dms_full_load_task.py
• airflow/providers/amazon/aws/example_dags/example_emr.py
• airflow/providers/amazon/aws/example_dags/example_sagemaker.py
• airflow/providers/amazon/aws/example_dags/example_eks_with_nodegroup_in_one_step.py
• airflow/providers/amazon/aws/example_dags/example_dynamodb_to_s3.py
• airflow/providers/amazon/aws/example_dags/example_s3_to_sftp.py
• airflow/providers/amazon/aws/example_dags/example_local_to_s3.py
• airflow/providers/amazon/aws/example_dags/example_s3_bucket.py
• airflow/providers/amazon/aws/example_dags/example_google_api_to_s3_transfer_basic.py
• airflow/providers/amazon/aws/example_dags/example_redshift_cluster.py
• airflow/providers/amazon/aws/example_dags/example_lambda.py
• airflow/providers/amazon/aws/example_dags/example_s3_to_redshift.py
• airflow/providers/amazon/aws/example_dags/example_imap_attachment_to_s3.py
• airflow/providers/amazon/aws/example_dags/example_ecs_fargate.py
• airflow/providers/amazon/aws/example_dags/example_rds.py
• airflow/providers/amazon/aws/example_dags/example_eks_with_fargate_in_one_step.py
• airflow/providers/amazon/aws/example_dags/example_batch.py
• airflow/providers/amazon/aws/example_dags/example_datasync_2.py
• airflow/providers/amazon/aws/example_dags/example_dynamodb_to_s3_segmented.py
• airflow/providers/amazon/aws/example_dags/example_ecs_ec2.py
• airflow/providers/amazon/aws/example_dags/example_eks_with_fargate_profile.py
• airflow/providers/amazon/aws/example_dags/example_redshift_to_s3.py
• airflow/providers/amazon/aws/example_dags/example_redshift_sql.py
• airflow/providers/amazon/aws/example_dags/example_sns.py
• airflow/providers/amazon/aws/example_dags/example_google_api_to_s3_transfer_advanced.py
• airflow/providers/amazon/aws/example_dags/example_datasync_1.py
• airflow/providers/amazon/aws/example_dags/example_glacier_to_gcs.py
• airflow/providers/amazon/aws/example_dags/example_redshift_data_execute_sql.py
• airflow/providers/amazon/aws/example_dags/example_eks_templated.py
• airflow/providers/amazon/aws/example_dags/example_sftp_to_s3.py
• airflow/providers/amazon/aws/example_dags/example_salesforce_to_s3.py

[ferruzzi]

Quick update, the following are not checked off above:

• airflow/providers/amazon/aws/example_dags/example_emr.py

  • @syedahsn is working on this one right now.

• airflow/providers/amazon/aws/example_dags/example_eks_templated.py

  • All of the operators in this DAG are tested in other system tests which have already been converted, so there is no added benefit to converting this one. It was initially a demo of how to use Jinja templating that I wrote and contributed when I was learning how to use it, so I do think it still has value as an example DAG. Should we delete this file, move it somewhere else, or just leave it? I feel like if we keep it, then “Example DAGs” is exactly where it should live. It uses AWS operators, but it’s specifically an AWS example (if that makes sense?) so maybe move it to a more generic location?

All of the remaining files were deemed out of scope for the current conversion project since they require external services/accounts/servers/etc (for example: requires a MongoDB account, a GCP account, hosting an SFTP server, etc) and we won’t be able to commit to running/maintaining them on a regular basis. We may add them at a later date, but they won’t be part of this big push.

• airflow/providers/amazon/aws/example_dags/example_s3_to_sftp.py
• airflow/providers/amazon/aws/example_dags/example_sftp_to_s3.py
• airflow/providers/amazon/aws/example_dags/example_imap_attachment_to_s3.py
  • I think this one goes here because it needed an SMTP server configured but I’m not certain?
• airflow/providers/amazon/aws/example_dags/example_glacier_to_gcs.py
• airflow/providers/amazon/aws/example_dags/example_salesforce_to_s3.py

[potiuk]

Moving the example dags in tests directory will not make the CI run them so we are safe on that side. Though I am wondering if we want to have example dags in this directory. That would mix system tests and example dags together. I would be actually inclined to have a subdirectory (e.g. example_dags) just for clarification purposes. WDYT?

Subdir is fine.