[BUG] Data corrupted in the timestamp field to 1970-01-01 19:45:30.000 after subsequent upsert run
See original GitHub issueTips before filing an issue
-
Have you gone through our FAQs?
-
Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
-
If you have triaged this as a bug, then file an issue directly.
Describe the problem you faced
I am using hudi as a part of Glue jobs to manage data mutations in our data lake. The only modification that is being applied during the ETL from raw to bronze (where hudi is involved) is introduction of timestamp field that transforms date in seconds field from raw (ie 1602297930 -> 2020-10-10 02:45:30.000). I have noticed that for some entities, after subsequent ingestions, the timestamp field value becomes corrupted (even for the records that are not present in the update). I managed to isolate the bulk_insert as a source of the issue. The timestamp field gets reset to 1970-01-01 19:45:30.000. Just to note, when insert is being used in place of bulk_insert for the initial ingest, the issue is not occurring.
To Reproduce
Steps to reproduce the behavior: Unfortunately it is pretty tricky to reproduce and it is only happening to 2 types of entities out of 16 we are using. We are moving stripe data into the data lake and the entities that do get issues are (https://stripe.com/docs/api/invoices and https://stripe.com/docs/api/customers
-
Create AWS Glue ETL job with hudi config
hudi_options = { "hoodie.table.create.schema": "default", "hoodie.table.name": tableName, "hoodie.datasource.write.recordkey.field": "id", "hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator", "hoodie.datasource.write.partitionpath.field": "field_1, field_2", "hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor", "hoodie.datasource.hive_sync.partition_fields": "field_1, field_2", 'hoodie.cleaner.policy': 'KEEP_LATEST_COMMITS', 'hoodie.cleaner.commits.retained': 10, "hoodie.datasource.write.table.name": tableName, "hoodie.datasource.write.operation": "bulk_insert", "hoodie.parquet.compression.codec": "snappy", "hoodie.datasource.hive_sync.enable": "true", "hoodie.datasource.hive_sync.use_jdbc": "false", 'hoodie.datasource.hive_sync.support_timestamp': "true", "hoodie.datasource.write.precombine.field": "lastupdate", "hoodie.datasource.hive_sync.database": f"{database_name}", "hoodie.datasource.hive_sync.table": tableName, }
The transformation of the raw data looks as following:sparkDF = dfc.select(list(dfc.keys())[0]).toDF() transformed = sparkDF.select("*", from_unixtime(col("created")).cast("timestamp").alias("created_ts")) resolvechoice4 = DynamicFrame.fromDF(transformed, glueContext, "transformedDF") return DynamicFrameCollection({"CustomTransform0": resolvechoice4}, glueContext)
-
Write data to data lake
sparkDF.write.format("hudi").options(**hudi_options).mode("overwrite").save( basePath )
-
Update the previously mentioned hudi config with write operation to become upsert. Update spark mode to append
-
Trigger Glue job again with updates
Expected behavior
I expect the data to remain consistent and the fields to represent the relevant info
Environment Description
-
Hudi version : 0.9.0
-
Spark version : 3.1
-
Hive version : Using Glue data catalogue as hive metastore
-
Hadoop version : N/A
-
Glue version: 3
-
Storage (HDFS/S3/GCS…) : S3
-
Running on Docker? (yes/no) :
Additional context
Add any other context about the problem here.
Stacktrace
There is no error
Issue Analytics
- State:
- Created 2 years ago
- Comments:14 (7 by maintainers)
I’ve tried it for BULK_INSERT and UPSERT as well, bot nothing works for me.
I prepared sample test job of that so you can reproduce it or just review it (I hope I just misconfigured it 😀). Upsert operation corrupts timestamps, no matter if I set this option or not - I tested it using DataFrame writer and SQL MERGE INTO.
AWS Glue 2.0(Spark 2.4) Hudi 0.10.1
https://github.com/apache/hudi/issues/5469 The 0.11.0 version fixed it.