Some questions about using hudi

Background

1. I am using non-partitioned copy on write hoodie table.
2. I am using Datasource API for bulk insert.
3. I am using spark streaming for incremental upsert( insert, update, delete)

Questions

1. Which parameter controls the parquet file size? I had set the following parameter but not working. There are lots of files with size 1.x MB after I bulk insert into hoodie table. .option("hoodie.upsert.shuffle.parallelism","200") .option("hoodie.insert.shuffle.parallelism", "100") .option("hoodie.upsert.shuffle.parallelism", "100") .option(HoodieStorageConfig.PARQUET_FILE_MAX_BYTES, 256 * 1024 * 1024) .option(HoodieCompactionConfig.PARQUET_SMALL_FILE_LIMIT_BYTES, 128 * 1024 * 1024)
2. For incremtal upserts, My workload is about 1000Record per batch. But It costs 2 mins , and most of the time is spent on HoodieBloomIndex.loadInvolvedFiles. The stage have so many parallel. Which parameter controls the parallel.

[n3nash]

There isn’t any generic recommended spark duration, it depends on a lot of factors such as amount of data ingested, parallelism of your job, how long it takes for you to write a 256 MB parquet file in your cluster etc which is best controlled by the clients. Having said that, here is what I’m saying :

1. Using bulkInsert() -> Depending on the spread of your data, this may create some small files. The bulkInsert() API does NOT do small file sizing, so in a scenario where you keep performing bulkInserts() on a dataset, you will end up creating small files.
2. Using upsert() -> This API takes a bunch of inserts and updates, applies the updates to existing data in files and pads the inserts to existing small files. If the number of inserts << the number of small files, it will take a long time for all the small files to be padded enough to reach 128MB file size.

So in summary, if you are using bulkInsert() repeatedly, you can see the side effect. If you are using upsert() after a bunch of small files are already lying around in your dataset, depending on how many inserts you are performing every batch, it will take sometime for all small files to be padded. Hope this makes things more clear.

[louisliu318]

@n3nash Thanks for your tips. I will do some tests on our cluster.