How to parametrise the dataform schedule

We are exploring dataform to integrate it with Airflow. We are not able to figure out options to pass the parameterized value to the schedule. The most generic use case could be like performing insert as select.

insert into new_table select * from source_dat where source_dat.created_at> ‘$create_At’ definitions/new_table.sqlx config { type: "table" } select * from source_dat

– We wanted to pass the value for create_at at scheduled time.

[mohammadskhan]

@BenBirt Thanks for your response, the incremental could only satisfy the scenarios when source data is time-series and contiguous. In our case, the data is time-series but it is very much common the x+1 hours data could come before x hours data. Hence chances of missing intervals in the dest table will be high. Is there any way to avoid the same supporting below options? Backfilling as & when required Concurrent execution for different interval

[astorrs]

At my company we’re also incredibly excited about Dataform and cannot wait to use it with Airflow. Similarly to @mohammadskhan we do generally implement DAGs that need to run parametrized queries. Dataform would be perfect to finally stop the sould crushing issues that come from running SQL through distributed files.

Is there are feature in consideration to parametrize runs? E.g. dataform run --dataform.projectConfig.vars.run_date="'2021-02-12'". Or just have placeholders in the dataform.json for that matter?

https://github.com/dataform-co/dataform/pull/1187 looks like it might be in the next release 👍