[SUPPORT] Avoid UPSERT unchanged records from source

Problem

When the source data set has unchanged rows, Hudi will upsert the target table rows and include those records in the new commit. If you have a CDC/incremental logic where you might have identical records from previous insert, new records, and changed records. Hudi would upsert all new, changed and unchanged records - and they would all be part of a new commit.

Now when you want to query increments, the result will include lot of unnecessary (unchanged) rows as well. I would like to avoid that. Is there a way to somehow drop unchanged rows from source?

To Reproduce

Steps to reproduce the behavior:

1. Fully load Hudi table

Target example:

---------------------------------------------------------------------
|     row_key    |     att_1      |      att_2     |    commit      |
---------------------------------------------------------------------
|        1       |      1_1       |       1_2      |        0       |
---------------------------------------------------------------------
|        2       |      2_1       |       2_2      |        0       |
---------------------------------------------------------------------

1. Incrementally upsert new data set (Incremental data set should include unchanged records)

Incremental data:

----------------------------------------------------
|     row_key    |     att_1      |      att_2     |  
----------------------------------------------------
|        1       |      1_1       |       1_2      |
----------------------------------------------------
|        2       |      2_1       |    changed     |
----------------------------------------------------
|        3       |      3_1       |       3_2      |
----------------------------------------------------
|        4       |      4_1       |       4_2      |
----------------------------------------------------

1. Incrementally query Hudi table for the latest commit

Target example:

---------------------------------------------------------------------
|     row_key    |     att_1      |      att_2     |    commit      |
---------------------------------------------------------------------
|        1       |      1_1       |       1_2      |        1       |
---------------------------------------------------------------------
|        2       |      2_1       |    changed     |        1       |
---------------------------------------------------------------------
|        3       |      3_1       |       3_2      |        1       |
---------------------------------------------------------------------
|        4       |      4_1       |       4_2      |        1       |
---------------------------------------------------------------------

Expected behavior

Target example:

---------------------------------------------------------------------
|     row_key    |     att_1      |      att_2     |    commit      |
---------------------------------------------------------------------
|        1       |      1_1       |       1_2      |        0       |
---------------------------------------------------------------------
|        2       |      2_1       |    changed     |        1       |
---------------------------------------------------------------------
|        3       |      3_1       |       3_2      |        1       |
---------------------------------------------------------------------
|        4       |      4_1       |       4_2      |        1       |
---------------------------------------------------------------------

Environment Description

• Hudi version : 0.5.3
• Spark version : 2.4.5
• Storage (HDFS/S3/GCS…) : S3

Thank you in advance!

[sleapfish]

@nsivabalan That is correct! But maybe if you have control over source but let’s say that you do something like extracting data from it with a rolling window of -3 days. And there is a case where some of the records from -3 days could change, but most of the records wouldn’t. I want to commit only changed/new records to target Hudi table.

When I do incremental query I don’t want all 3 days worth of data in it, even tho only small portion of it actually changed

[victorcadena]

One question, if the persisted record has the same Ordering Value as the Incoming Record, the record is updated, right?

https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/DefaultHoodieRecordPayload.java#L139

Is there an option to change that behavior? For me, if the ordering is the same it would be more useful to leave the old value as I don’t want to waste resources merging something unchanged. Is that a possibility?

@nsivabalan