feature: Support to run jobs with --full-refresh option from airflow

Feature scope

Web UI

Description

Currently, I can add a job with tasks e.g meltano job add <job_name> --tasks "<tap_name> <target_name>". But when I want to run a job with full-refresh option, I get an error The job has invalid task. Adding the support of full-refresh will help us to schedule the job from airflow. In production environments, we don’t have access into the container (meltano deployed in kubernetes) to run CLI. Hence, it’s difficult to run a job with full-refresh option.

[aaronsteers]

@tayloramurphy - Yeah, I think the “C” option would make a lot of sense. Basically: “assume each line is a string to be passed to run unless it’s prefixed by another command”.

This has the nice benefit of being neatly copy-pasteable.

# Before the feature, we have to run them separately:
meltano run tap-github target-snowflake
meltano invoke dbt:run --select=foo+  # This doesn't work as a job task
meltano run dbt:test

# After the feature, we can copy-paste each line in as a task:
meltano job create my_new_job --tasks "[tap-github target-snowflake, invoke: dbt:run --select=foo+, dbt:test]
meltano run my_new_job

Which creates this yaml:

  tasks:
  - tap-github target-snowflake
  - invoke: dbt:run --select=foo+
  - dbt:test

Which would be equivalent to:

  tasks:
  - run: tap-github target-snowflake
  - invoke: dbt:run --select=foo+
  - run: dbt:test

[tayloramurphy]

@aaronsteers yep - I quite like that