[CT-1356] [Feature] incremental models' merge conditional update

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

In some databases (namely snowflake and postgres), it is possible to add a condition to a merge update clause: when matched and <condition> then ... I want to allow to add this kind of condition to incremental models (by making it an incremental model configuration).

Describe alternatives you’ve considered

A considerable alternative would be to join the records in the incremental run, with their destination records, to check for which rows the condition is not satisfied, and omit these rows. such solution would be less performant and will make the model more complicated to understand.

Who will this benefit?

For example when there is a scenario of out of order data that is arbitrarily inserted into the source table of a model, this feature allow us to omit this data by using a condition like: DBT_INTERNAL_SOURCE.timestamp > DBT_INTERNAL_DEST.timestamp.

Are you interested in contributing this feature?

yes

Anything else?

I have forks with suggested implementation: https://github.com/Yotamho/dbt-core https://github.com/Yotamho/dbt-snowflake

[tphillip33]

In reviewing this implementation. For my purpose, I think I would rather simply override default__get_merge_sql, to accept a new config setting of “merge_update_only_when_columns_different” (or something) and append it to the “when matched {merge_update_only_when_columns_different code here} then update set…”

I will try that and see how that goes.

[jtcohen6]

After giving this another read through, I see significant overlap between this issue, and one I just responded to yesterday: https://github.com/dbt-labs/dbt-core/issues/6415#issuecomment-1348827516

Two points that carry over in particular:

• As a general rule, we want out-of-the-box materialization behavior to be as simple & easy to reason about as possible. If given the choice between two approaches, with functionally identical outcomes, we should opt for the approach that keeps more of the business logic within the model SQL select statement.
• At the same time, we want it to be possible to “register” a custom incremental strategy, when you want merge to work differently for certain models. There’s a new & better pattern for doing this, starting in v1.3 (but we still need to document it! https://github.com/dbt-labs/docs.getdbt.com/issues/1761). You could define (e.g.) a macro named get_incremental_merge_conditional_update_sql, and set incremental_strategy: 'merge_conditional_update' in your model config, along with any other configs that would be relevant to templating your custom merge statement. All that, without having to copy-paste-edit the incremental materialization, or defining one of your own.