[SUPPORT] Trino hudi query slow when compared to parquet
See original GitHub issueDescribe the problem you faced
Presto query performance with hudi table takes ~2x extra time when compared to parquet for simple query . data stored in s3. hudi metadata store enabled. note, spark sql much faster with hudi dataset.
parquet query first attempt: 23 secs parquet query second attempt: 0.6 secs hudi query first attempt: 40 sec hudi query second attempt: 40 sec data size : 200gb
also, parquet query faster after first run. we are in poc stage to migrate parquet to Hudi data lake. it is one of the blocker. help appreciated.
Environment Description
-
Hudi version : master (4a5683d54a3910e1c860f26b291e1e99931b5949)
-
Spark version : 2.4.0
-
Hive version : 2.3.3
-
Hadoop version :2.8.5
-
Storage (HDFS/S3/GCS…) : S3
-
Running on Docker? (yes/no) : yes, presto runs on k8
Additional context Presto (trino 352) runs on k8. hudi bundle added.
Stacktrace
parquet Query (first attempt)
======================================
presto> select y, m, d, h, count(*) from hive.parquet_warehouse.sample_data where y=2021 and m=2 and d=20 and h between 0 and 4 group by y, m, d, h;
Query 20210227_043723_00000_52ygg, RUNNING, 8 nodes, 6,122 splits
0:23 [ 450M rows, 0B] [19.5M rows/s, 0B/s] [====================================>>> ] 85%
STAGES ROWS ROWS/s BYTES BYTES/s QUEUED RUN DONE
0.........R 0 0 0B 0B 0 16 0
1.......R 4.81K 207 685K 29.6K 0 256 0
2.....R 450M 19.4M 0B 0B 524 80 5246
y | m | d | h | _col4
------+---+----+---+-----------
2021 | 2 | 20 | 4 | 109245879
2021 | 2 | 20 | 0 | 76442859
2021 | 2 | 20 | 1 | 94755366
2021 | 2 | 20 | 2 | 101223162
2021 | 2 | 20 | 3 | 103258290
(5 rows)
Query 20210227_043723_00000_52ygg, FINISHED, 8 nodes
Splits: 6,183 total, 6,183 done (100.00%)
0:23 [485M rows, 0B] [20.9M rows/s, 0B/s]
parquet query (second attempt)
======================================
presto> select y, m, d, h, count(*) from hive.parquet_warehouse.sample_data where y=2021 and m=2 and d=20 and h between 0 and 4 group by y, m, d, h;
Query 20210227_043810_00001_52ygg, RUNNING, 10 nodes, 6,215 splits
0:05 [ 475M rows, 0B] [95.6M rows/s, 0B/s] [=======================================>>>] 91%
STAGES ROWS ROWS/s BYTES BYTES/s QUEUED RUN DONE
0.........R 0 0 0B 0B 0 16 0
1.......R 4.81K 966 686K 138K 0 320 0
2.....R 475M 95.3M 0B 0B 120 94 5665
y | m | d | h | _col4
------+---+----+---+-----------
2021 | 2 | 20 | 3 | 103258290
2021 | 2 | 20 | 2 | 101223162
2021 | 2 | 20 | 0 | 76442859
2021 | 2 | 20 | 1 | 94755366
2021 | 2 | 20 | 4 | 109245879
(5 rows)
Query 20210227_043810_00001_52ygg, FINISHED, 10 nodes
Splits: 6,215 total, 6,215 done (100.00%)
0:05 [485M rows, 0B] [94.2M rows/s, 0B/s]
hudi query (first attempt)
==========================================
presto> select y, m, d, h, count(*) from hive.hudi_warehouse.sample_data group by y, m, d, h;
Query 20210227_042722_00016_9dket, RUNNING, 10 nodes, 8,443 splits
0:40 [ 473M rows, 0B] [ 12M rows/s, 0B/s] [ <=> ]
STAGES ROWS ROWS/s BYTES BYTES/s QUEUED RUN DONE
0.........R 0 0 0B 0B 0 16 0
1.......R 3.26K 82 464K 11.7K 0 320 0
2.....S 473M 12M 0B 0B 43 18 8046
y | m | d | h | _col4
------+---+----+---+-----------
2021 | 2 | 20 | 1 | 94755289
2021 | 2 | 20 | 4 | 109245869
2021 | 2 | 20 | 0 | 76442808
2021 | 2 | 20 | 2 | 101223154
2021 | 2 | 20 | 3 | 103258285
(5 rows)
Query 20210227_042722_00016_9dket, FINISHED, 10 nodes
Splits: 8,572 total, 8,572 done (100.00%)
0:40 [485M rows, 0B] [12.1M rows/s, 0B/s]
presto logs for hudi Query
================================
2021-02-27T04:27:23.112Z DEBUG dispatcher-query-24 io.prestosql.execution.QueryStateMachine Query 20210227_042722_00016_9dket is QUEUED
2021-02-27T04:27:23.114Z DEBUG dispatcher-query-28 io.prestosql.execution.QueryStateMachine Query 20210227_042722_00016_9dket is WAITING_FOR_RESOURCES
2021-02-27T04:27:23.167Z DEBUG Query-20210227_042722_00016_9dket-664 io.prestosql.plugin.hive.metastore.thrift.ThriftHiveMetastoreClient Invocation of get_table_req(req=GetTableRequest(dbName:hudi_warehouse, tblName:sample_data_cluster_2, capabilities:ClientCapabilities(values:[INSERT_ONLY_TABLES]))) succeeded in 51.56ms
2021-02-27T04:27:23.194Z DEBUG dispatcher-query-25 io.prestosql.execution.QueryStateMachine Query 20210227_042722_00016_9dket is DISPATCHING
2021-02-27T04:27:23.194Z DEBUG dispatcher-query-26 io.prestosql.execution.QueryStateMachine Query 20210227_042722_00016_9dket is PLANNING
2021-02-27T04:27:23.276Z DEBUG Query-20210227_042722_00016_9dket-667 io.prestosql.plugin.hive.metastore.thrift.ThriftHiveMetastoreClient Invocation of get_partition_names_ps(db_name='hudi_warehouse', tbl_name='sample_data_cluster_2', part_vals=[, , , ], max_parts=-1) succeeded in 60.66ms
2021-02-27T04:27:23.364Z DEBUG hive-hive-16 io.prestosql.plugin.hive.metastore.thrift.ThriftHiveMetastoreClient Invocation of get_partitions_by_names(db_name='hudi_warehouse', tbl_name='sample_data_cluster_2', names=[y=2021/m=2/d=20/h=4, y=2021/m=2/d=20/h=3, y=2021/m=2/d=20/h=2, y=2021/m=2/d=20/h=1, y=2021/m=2/d=20/h=0]) succeeded in 58.46ms
2021-02-27T04:27:23.369Z DEBUG query-execution-13 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.0 is PLANNED
2021-02-27T04:27:23.369Z DEBUG query-execution-13 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.1 is PLANNED
2021-02-27T04:27:23.373Z DEBUG query-execution-14 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.2 is PLANNED
2021-02-27T04:27:23.373Z DEBUG dispatcher-query-25 io.prestosql.execution.QueryStateMachine Query 20210227_042722_00016_9dket is STARTING
2021-02-27T04:27:23.378Z DEBUG query-execution-16 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.2 is SCHEDULING
2021-02-27T04:27:23.379Z DEBUG query-execution-14 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.1 is SCHEDULING
2021-02-27T04:27:23.420Z DEBUG dispatcher-query-26 io.prestosql.execution.QueryStateMachine Query 20210227_042722_00016_9dket is RUNNING
2021-02-27T04:27:23.425Z INFO hive-hive-19 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Reading hoodie metadata from path s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.426Z INFO hive-hive-19 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.427Z DEBUG query-execution-14 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.1 is SCHEDULED
2021-02-27T04:27:23.427Z INFO hive-hive-19 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.428Z DEBUG query-execution-14 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.0 is SCHEDULING
2021-02-27T04:27:23.428Z DEBUG query-execution-14 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.1 is RUNNING
2021-02-27T04:27:23.429Z INFO hive-hive-17 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Reading hoodie metadata from path s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.429Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.430Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.432Z INFO hive-hive-18 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Reading hoodie metadata from path s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.433Z INFO hive-hive-16 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Reading hoodie metadata from path s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.433Z INFO hive-hive-18 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.433Z INFO hive-hive-16 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.434Z INFO hive-hive-18 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.434Z DEBUG query-execution-14 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.0 is SCHEDULED
2021-02-27T04:27:23.437Z DEBUG query-execution-16 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.0 is RUNNING
2021-02-27T04:27:23.437Z INFO hive-hive-16 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.473Z INFO hive-hive-19 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.473Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.474Z INFO hive-hive-16 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.490Z INFO hive-hive-18 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.502Z INFO hive-hive-16 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.503Z INFO hive-hive-16 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Found a total of 1 groups
2021-02-27T04:27:23.503Z INFO hive-hive-19 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.503Z INFO hive-hive-16 org.apache.hudi.common.table.view.FileSystemViewManager Creating InMemory based view for basePath s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.503Z INFO hive-hive-19 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Found a total of 1 groups
2021-02-27T04:27:23.503Z INFO hive-hive-19 org.apache.hudi.common.table.view.FileSystemViewManager Creating InMemory based view for basePath s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.523Z INFO hive-hive-19 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED], [20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED], [20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.523Z INFO hive-hive-19 org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.529Z INFO hive-hive-16 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED], [20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED], [20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.530Z INFO hive-hive-16 org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.539Z INFO hive-hive-18 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.540Z INFO hive-hive-18 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Found a total of 1 groups
2021-02-27T04:27:23.540Z INFO hive-hive-18 org.apache.hudi.common.table.view.FileSystemViewManager Creating InMemory based view for basePath s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.541Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.542Z INFO hive-hive-17 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Found a total of 1 groups
2021-02-27T04:27:23.542Z INFO hive-hive-17 org.apache.hudi.common.table.view.FileSystemViewManager Creating InMemory based view for basePath s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.561Z INFO hive-hive-19 org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending clustering operations
2021-02-27T04:27:23.561Z INFO hive-hive-18 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED], [20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED], [20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.561Z INFO hive-hive-19 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.562Z INFO hive-hive-17 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED], [20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED], [20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.562Z INFO hive-hive-18 org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.562Z INFO hive-hive-19 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.562Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.567Z INFO hive-hive-16 org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending clustering operations
2021-02-27T04:27:23.568Z INFO hive-hive-16 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.569Z INFO hive-hive-16 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.598Z INFO hive-hive-18 org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending clustering operations
2021-02-27T04:27:23.598Z INFO hive-hive-18 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.599Z INFO hive-hive-17 org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending clustering operations
2021-02-27T04:27:23.599Z INFO hive-hive-18 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.600Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.601Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.604Z INFO hive-hive-19 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.606Z INFO hive-hive-16 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.619Z INFO hive-hive-19 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.619Z INFO hive-hive-19 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.620Z INFO hive-hive-19 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.620Z INFO hive-hive-16 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.620Z INFO hive-hive-16 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.621Z INFO hive-hive-16 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.638Z INFO hive-hive-18 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.639Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:23.652Z INFO hive-hive-18 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.653Z INFO hive-hive-18 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.655Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:23.655Z INFO hive-hive-18 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.656Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.657Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.659Z INFO hive-hive-16 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:23.670Z INFO hive-hive-19 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:23.674Z INFO hive-hive-16 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.687Z INFO hive-hive-19 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.695Z INFO hive-hive-18 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:23.696Z INFO hive-hive-16 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[0000000000000__deltacommit__COMPLETED], [20210227020224__deltacommit__COMPLETED], [20210227021755__deltacommit__COMPLETED], [20210227024056__deltacommit__COMPLETED], [20210227030430__deltacommit__COMPLETED], [20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.697Z INFO hive-hive-16 org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.700Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:23.710Z INFO hive-hive-19 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[0000000000000__deltacommit__COMPLETED], [20210227020224__deltacommit__COMPLETED], [20210227021755__deltacommit__COMPLETED], [20210227024056__deltacommit__COMPLETED], [20210227030430__deltacommit__COMPLETED], [20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.710Z INFO hive-hive-19 org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.711Z INFO hive-hive-18 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.714Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.729Z INFO hive-hive-18 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[0000000000000__deltacommit__COMPLETED], [20210227020224__deltacommit__COMPLETED], [20210227021755__deltacommit__COMPLETED], [20210227024056__deltacommit__COMPLETED], [20210227030430__deltacommit__COMPLETED], [20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.730Z INFO hive-hive-18 org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.732Z INFO hive-hive-17 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[0000000000000__deltacommit__COMPLETED], [20210227020224__deltacommit__COMPLETED], [20210227021755__deltacommit__COMPLETED], [20210227024056__deltacommit__COMPLETED], [20210227030430__deltacommit__COMPLETED], [20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.733Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to read 0 instants, 0 replaced file groups
2021-02-27T04:27:23.737Z INFO hive-hive-16 org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending clustering operations
2021-02-27T04:27:23.737Z INFO hive-hive-16 org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file system view for partition (files)
2021-02-27T04:27:23.746Z INFO hive-hive-19 org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending clustering operations
2021-02-27T04:27:23.747Z INFO hive-hive-19 org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file system view for partition (files)
2021-02-27T04:27:23.767Z INFO hive-hive-18 org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending clustering operations
2021-02-27T04:27:23.767Z INFO hive-hive-18 org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file system view for partition (files)
2021-02-27T04:27:23.771Z INFO hive-hive-17 org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending clustering operations
2021-02-27T04:27:23.772Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file system view for partition (files)
2021-02-27T04:27:23.793Z INFO hive-hive-16 org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in partition (files) =7, Time taken =19
2021-02-27T04:27:23.795Z INFO hive-hive-16 org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding file-groups for partition :files, #FileGroups=1
2021-02-27T04:27:23.795Z INFO hive-hive-16 org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView: NumFiles=7, NumFileGroups=1, FileGroupsCreationTime=2, StoreTimeTaken=0
2021-02-27T04:27:23.795Z INFO hive-hive-16 org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load partition (files) =58
2021-02-27T04:27:23.795Z INFO hive-hive-16 org.apache.hudi.common.table.view.AbstractTableFileSystemView Pending Compaction instant for (FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='files', fileId='7e172a24-c399-4f87-a9a7-75b3d2879e0b-0'}, baseCommitTime=0000000000000, baseFile='HoodieBaseFile{fullPath=s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile, fileLen=5696, BootstrapBaseFile=null}', logFiles='[HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=25302}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=23561}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=23243}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=21732}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553', fileLen=27253}]'}) is :Optional.empty
2021-02-27T04:27:23.805Z INFO hive-hive-19 org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in partition (files) =7, Time taken =20
2021-02-27T04:27:23.807Z INFO hive-hive-19 org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding file-groups for partition :files, #FileGroups=1
2021-02-27T04:27:23.807Z INFO hive-hive-19 org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView: NumFiles=7, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=1
2021-02-27T04:27:23.807Z INFO hive-hive-19 org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load partition (files) =60
2021-02-27T04:27:23.807Z INFO hive-hive-19 org.apache.hudi.common.table.view.AbstractTableFileSystemView Pending Compaction instant for (FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='files', fileId='7e172a24-c399-4f87-a9a7-75b3d2879e0b-0'}, baseCommitTime=0000000000000, baseFile='HoodieBaseFile{fullPath=s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile, fileLen=5696, BootstrapBaseFile=null}', logFiles='[HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=25302}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=23561}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=23243}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=21732}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553', fileLen=27253}]'}) is :Optional.empty
2021-02-27T04:27:23.815Z INFO hive-hive-16 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED], [20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED], [20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.827Z INFO hive-hive-18 org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in partition (files) =7, Time taken =22
2021-02-27T04:27:23.828Z INFO hive-hive-19 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED], [20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED], [20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.829Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in partition (files) =7, Time taken =17
2021-02-27T04:27:23.829Z INFO hive-hive-18 org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding file-groups for partition :files, #FileGroups=1
2021-02-27T04:27:23.829Z INFO hive-hive-18 org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView: NumFiles=7, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
2021-02-27T04:27:23.829Z INFO hive-hive-18 org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load partition (files) =62
2021-02-27T04:27:23.830Z INFO hive-hive-18 org.apache.hudi.common.table.view.AbstractTableFileSystemView Pending Compaction instant for (FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='files', fileId='7e172a24-c399-4f87-a9a7-75b3d2879e0b-0'}, baseCommitTime=0000000000000, baseFile='HoodieBaseFile{fullPath=s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile, fileLen=5696, BootstrapBaseFile=null}', logFiles='[HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=25302}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=23561}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=23243}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=21732}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553', fileLen=27253}]'}) is :Optional.empty
2021-02-27T04:27:23.831Z INFO hive-hive-17 org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding file-groups for partition :files, #FileGroups=1
2021-02-27T04:27:23.831Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView: NumFiles=7, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=1
2021-02-27T04:27:23.831Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load partition (files) =59
2021-02-27T04:27:23.831Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView Pending Compaction instant for (FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='files', fileId='7e172a24-c399-4f87-a9a7-75b3d2879e0b-0'}, baseCommitTime=0000000000000, baseFile='HoodieBaseFile{fullPath=s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile, fileLen=5696, BootstrapBaseFile=null}', logFiles='[HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=25302}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=23561}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=23243}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=21732}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553', fileLen=27253}]'}) is :Optional.empty
2021-02-27T04:27:23.834Z INFO hive-hive-16 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[0000000000000__deltacommit__COMPLETED], [20210227020224__deltacommit__COMPLETED], [20210227021755__deltacommit__COMPLETED], [20210227024056__deltacommit__COMPLETED], [20210227030430__deltacommit__COMPLETED], [20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.835Z INFO hive-hive-16 org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata table is disabled.
2021-02-27T04:27:23.835Z INFO hive-hive-16 org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file system view for partition (y=2021/m=2/d=20/h=4)
2021-02-27T04:27:23.849Z INFO hive-hive-18 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED], [20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED], [20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.849Z INFO hive-hive-19 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[0000000000000__deltacommit__COMPLETED], [20210227020224__deltacommit__COMPLETED], [20210227021755__deltacommit__COMPLETED], [20210227024056__deltacommit__COMPLETED], [20210227030430__deltacommit__COMPLETED], [20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.849Z INFO hive-hive-19 org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata table is disabled.
2021-02-27T04:27:23.849Z INFO hive-hive-17 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED], [20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED], [20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:23.850Z INFO hive-hive-19 org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file system view for partition (y=2021/m=2/d=20/h=3)
2021-02-27T04:27:23.869Z INFO hive-hive-17 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[0000000000000__deltacommit__COMPLETED], [20210227020224__deltacommit__COMPLETED], [20210227021755__deltacommit__COMPLETED], [20210227024056__deltacommit__COMPLETED], [20210227030430__deltacommit__COMPLETED], [20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.870Z INFO hive-hive-17 org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata table is disabled.
2021-02-27T04:27:23.870Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file system view for partition (y=2021/m=2/d=20/h=1)
2021-02-27T04:27:23.872Z INFO hive-hive-18 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[0000000000000__deltacommit__COMPLETED], [20210227020224__deltacommit__COMPLETED], [20210227021755__deltacommit__COMPLETED], [20210227024056__deltacommit__COMPLETED], [20210227030430__deltacommit__COMPLETED], [20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:23.873Z INFO hive-hive-18 org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata table is disabled.
2021-02-27T04:27:23.873Z INFO hive-hive-18 org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file system view for partition (y=2021/m=2/d=20/h=2)
2021-02-27T04:27:23.952Z INFO hive-hive-16 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:23.953Z INFO hive-hive-16 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.972Z INFO hive-hive-17 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:23.973Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.983Z INFO hive-hive-19 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:23.983Z INFO hive-hive-19 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.995Z INFO hive-hive-16 org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata base file from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile at instant 0000000000000
2021-02-27T04:27:23.995Z INFO hive-hive-18 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:23.995Z INFO hive-hive-18 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:23.996Z INFO hive-hive-16 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:23.997Z INFO hive-hive-16 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:24.034Z INFO hive-hive-17 org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata base file from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile at instant 0000000000000
2021-02-27T04:27:24.036Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:24.037Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:24.050Z INFO hive-hive-16 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:24.051Z INFO hive-hive-19 org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata base file from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile at instant 0000000000000
2021-02-27T04:27:24.052Z INFO hive-hive-19 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:24.053Z INFO hive-hive-19 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:24.068Z INFO hive-hive-18 org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata base file from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile at instant 0000000000000
2021-02-27T04:27:24.070Z INFO hive-hive-18 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:24.070Z INFO hive-hive-16 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:24.071Z INFO hive-hive-18 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:24.075Z INFO hive-hive-16 org.apache.hudi.common.util.collection.DiskBasedMap Spilling to file location /tmp/view_map/de468b42-7a11-47fb-a360-fb966c8f8b15 in host (100.97.190.93) with hostname (presto-coordinator-5fbd7bd6bd-bhx56)
2021-02-27T04:27:24.096Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:24.106Z INFO hive-hive-19 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:24.111Z INFO hive-hive-18 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:24.112Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553', fileLen=0}
2021-02-27T04:27:24.114Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:24.114Z INFO hive-hive-17 org.apache.hudi.common.util.collection.DiskBasedMap Spilling to file location /tmp/view_map/287eb55d-674d-4f7f-82c1-7a766c596e64 in host (100.97.190.93) with hostname (presto-coordinator-5fbd7bd6bd-bhx56)
2021-02-27T04:27:24.122Z INFO hive-hive-19 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:24.123Z INFO hive-hive-19 org.apache.hudi.common.util.collection.DiskBasedMap Spilling to file location /tmp/view_map/2a5b2fb5-5287-4b3b-8c9c-8ce4ebfed736 in host (100.97.190.93) with hostname (presto-coordinator-5fbd7bd6bd-bhx56)
2021-02-27T04:27:24.125Z INFO hive-hive-18 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:24.125Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553 at instant 20210227020224
2021-02-27T04:27:24.126Z INFO hive-hive-18 org.apache.hudi.common.util.collection.DiskBasedMap Spilling to file location /tmp/view_map/52ad2374-1441-48f2-8856-f4838ca8bec5 in host (100.97.190.93) with hostname (presto-coordinator-5fbd7bd6bd-bhx56)
2021-02-27T04:27:24.140Z INFO hive-hive-16 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=0}
2021-02-27T04:27:24.145Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553', fileLen=0}
2021-02-27T04:27:24.151Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553', fileLen=0}
2021-02-27T04:27:24.155Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=0}
2021-02-27T04:27:24.156Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553', fileLen=0}
2021-02-27T04:27:24.164Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553 at instant 20210227020224
2021-02-27T04:27:24.168Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553 at instant 20210227020224
2021-02-27T04:27:24.171Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553 at instant 20210227020224
2021-02-27T04:27:24.171Z INFO hive-hive-19 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=0}
2021-02-27T04:27:24.176Z INFO hive-hive-18 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=0}
2021-02-27T04:27:24.177Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183 at instant 20210227021755
2021-02-27T04:27:24.178Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.179Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=0}
2021-02-27T04:27:24.184Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=0}
2021-02-27T04:27:24.188Z INFO hive-hive-16 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.189Z INFO hive-hive-16 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.190Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=0}
2021-02-27T04:27:24.191Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=0}
2021-02-27T04:27:24.191Z INFO hive-hive-16 org.apache.hudi.common.util.collection.ExternalSpillableMap Estimated Payload size => 792
2021-02-27T04:27:24.198Z INFO hive-hive-16 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=0}
2021-02-27T04:27:24.200Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183 at instant 20210227021755
2021-02-27T04:27:24.200Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.204Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183 at instant 20210227021755
2021-02-27T04:27:24.204Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.207Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183 at instant 20210227021755
2021-02-27T04:27:24.208Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.213Z INFO hive-hive-19 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.213Z INFO hive-hive-19 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.215Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=0}
2021-02-27T04:27:24.216Z INFO hive-hive-19 org.apache.hudi.common.util.collection.ExternalSpillableMap Estimated Payload size => 792
2021-02-27T04:27:24.218Z INFO hive-hive-18 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.218Z INFO hive-hive-18 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.220Z INFO hive-hive-18 org.apache.hudi.common.util.collection.ExternalSpillableMap Estimated Payload size => 792
2021-02-27T04:27:24.223Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.223Z INFO hive-hive-19 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=0}
2021-02-27T04:27:24.223Z INFO hive-hive-17 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.228Z INFO hive-hive-18 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=0}
2021-02-27T04:27:24.233Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049 at instant 20210227024056
2021-02-27T04:27:24.233Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.235Z INFO hive-hive-17 org.apache.hudi.common.util.collection.ExternalSpillableMap Estimated Payload size => 792
2021-02-27T04:27:24.239Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=0}
2021-02-27T04:27:24.244Z INFO hive-hive-16 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.245Z INFO hive-hive-16 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.245Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=0}
2021-02-27T04:27:24.247Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=0}
2021-02-27T04:27:24.255Z INFO hive-hive-16 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=0}
2021-02-27T04:27:24.258Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049 at instant 20210227024056
2021-02-27T04:27:24.258Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.262Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049 at instant 20210227024056
2021-02-27T04:27:24.262Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.270Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=0}
2021-02-27T04:27:24.271Z INFO hive-hive-19 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.271Z INFO hive-hive-19 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.275Z INFO hive-hive-18 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.275Z INFO hive-hive-18 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.281Z INFO hive-hive-19 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=0}
2021-02-27T04:27:24.285Z INFO hive-hive-18 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=0}
2021-02-27T04:27:24.288Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=0}
2021-02-27T04:27:24.289Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049 at instant 20210227024056
2021-02-27T04:27:24.289Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.297Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=0}
2021-02-27T04:27:24.301Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=0}
2021-02-27T04:27:24.302Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088 at instant 20210227030430
2021-02-27T04:27:24.302Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.303Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.303Z INFO hive-hive-17 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.313Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=0}
2021-02-27T04:27:24.314Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088 at instant 20210227030430
2021-02-27T04:27:24.314Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.315Z INFO hive-hive-16 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.316Z INFO hive-hive-16 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.326Z INFO hive-hive-16 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=0}
2021-02-27T04:27:24.326Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088 at instant 20210227030430
2021-02-27T04:27:24.326Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.327Z INFO hive-hive-19 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.327Z INFO hive-hive-19 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.333Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=0}
2021-02-27T04:27:24.338Z INFO hive-hive-19 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=0}
2021-02-27T04:27:24.340Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=0}
2021-02-27T04:27:24.348Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088 at instant 20210227030430
2021-02-27T04:27:24.348Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.353Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093 at instant 20210227032808
2021-02-27T04:27:24.354Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.356Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=0}
2021-02-27T04:27:24.356Z INFO hive-hive-18 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.357Z INFO hive-hive-18 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.372Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.372Z INFO hive-hive-17 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.373Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093 at instant 20210227032808
2021-02-27T04:27:24.373Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.382Z INFO hive-hive-16 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.382Z INFO hive-hive-16 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.383Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=0}
2021-02-27T04:27:24.384Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Merging the final data blocks
2021-02-27T04:27:24.384Z INFO hive-hive-16 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.387Z INFO hive-hive-19 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.387Z INFO hive-hive-19 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.390Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Merging the final data blocks
2021-02-27T04:27:24.390Z INFO hive-hive-19 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.397Z INFO hive-hive-18 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=0}
2021-02-27T04:27:24.398Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=0}
2021-02-27T04:27:24.413Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=0}
2021-02-27T04:27:24.416Z INFO hive-hive-16 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.416Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093 at instant 20210227032808
2021-02-27T04:27:24.416Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.416Z INFO hive-hive-16 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.418Z INFO hive-hive-16 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of log files scanned => 5
2021-02-27T04:27:24.419Z INFO hive-hive-16 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner MaxMemoryInBytes allowed for compaction => 0
2021-02-27T04:27:24.419Z INFO hive-hive-16 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of entries in MemoryBasedMap in ExternalSpillableMap => 6
2021-02-27T04:27:24.419Z INFO hive-hive-16 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 4752
2021-02-27T04:27:24.419Z INFO hive-hive-16 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of entries in DiskBasedMap in ExternalSpillableMap => 0
2021-02-27T04:27:24.419Z INFO hive-hive-16 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Size of file spilled to disk => 0
2021-02-27T04:27:24.419Z INFO hive-hive-16 org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata log files from [s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093] at instant 20210227032808(dataset instant=20210227032808, metadata instant=20210227032808)
2021-02-27T04:27:24.422Z INFO hive-hive-19 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.422Z INFO hive-hive-19 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.425Z INFO hive-hive-19 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of log files scanned => 5
2021-02-27T04:27:24.425Z INFO hive-hive-19 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner MaxMemoryInBytes allowed for compaction => 0
2021-02-27T04:27:24.425Z INFO hive-hive-19 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of entries in MemoryBasedMap in ExternalSpillableMap => 6
2021-02-27T04:27:24.425Z INFO hive-hive-19 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 4752
2021-02-27T04:27:24.425Z INFO hive-hive-19 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of entries in DiskBasedMap in ExternalSpillableMap => 0
2021-02-27T04:27:24.426Z INFO hive-hive-19 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Size of file spilled to disk => 0
2021-02-27T04:27:24.426Z INFO hive-hive-19 org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata log files from [s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093] at instant 20210227032808(dataset instant=20210227032808, metadata instant=20210227032808)
2021-02-27T04:27:24.430Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093 at instant 20210227032808
2021-02-27T04:27:24.431Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.435Z INFO hive-hive-16 org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata read for key y=2021/m=2/d=20/h=4 took [open, baseFileRead, logMerge] [467, 16, 0] ms
2021-02-27T04:27:24.436Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.436Z INFO hive-hive-17 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.439Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Merging the final data blocks
2021-02-27T04:27:24.439Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.440Z INFO hive-hive-19 org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata read for key y=2021/m=2/d=20/h=3 took [open, baseFileRead, logMerge] [443, 14, 0] ms
2021-02-27T04:27:24.445Z INFO hive-hive-16 org.apache.hudi.metadata.BaseTableMetadata Listed file in partition from metadata: partition=y=2021/m=2/d=20/h=4, #files=761
2021-02-27T04:27:24.445Z INFO hive-hive-16 org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in partition (y=2021/m=2/d=20/h=4) =761, Time taken =493
2021-02-27T04:27:24.450Z INFO hive-hive-19 org.apache.hudi.metadata.BaseTableMetadata Listed file in partition from metadata: partition=y=2021/m=2/d=20/h=3, #files=699
2021-02-27T04:27:24.450Z INFO hive-hive-19 org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in partition (y=2021/m=2/d=20/h=3) =699, Time taken =467
2021-02-27T04:27:24.452Z INFO hive-hive-18 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.452Z INFO hive-hive-18 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.459Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.460Z INFO hive-hive-17 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.462Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Merging the final data blocks
2021-02-27T04:27:24.462Z INFO hive-hive-18 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:24.462Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of log files scanned => 5
2021-02-27T04:27:24.463Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner MaxMemoryInBytes allowed for compaction => 0
2021-02-27T04:27:24.463Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of entries in MemoryBasedMap in ExternalSpillableMap => 6
2021-02-27T04:27:24.463Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 4752
2021-02-27T04:27:24.463Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of entries in DiskBasedMap in ExternalSpillableMap => 0
2021-02-27T04:27:24.463Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Size of file spilled to disk => 0
2021-02-27T04:27:24.463Z INFO hive-hive-17 org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata log files from [s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093] at instant 20210227032808(dataset instant=20210227032808, metadata instant=20210227032808)
2021-02-27T04:27:24.480Z INFO hive-hive-17 org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata read for key y=2021/m=2/d=20/h=1 took [open, baseFileRead, logMerge] [491, 17, 0] ms
2021-02-27T04:27:24.487Z INFO hive-hive-18 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:24.487Z INFO hive-hive-18 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:24.491Z INFO hive-hive-17 org.apache.hudi.metadata.BaseTableMetadata Listed file in partition from metadata: partition=y=2021/m=2/d=20/h=1, #files=647
2021-02-27T04:27:24.491Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in partition (y=2021/m=2/d=20/h=1) =647, Time taken =519
2021-02-27T04:27:24.501Z INFO hive-hive-16 org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding file-groups for partition :y=2021/m=2/d=20/h=4, #FileGroups=761
2021-02-27T04:27:24.501Z INFO hive-hive-16 org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView: NumFiles=761, NumFileGroups=761, FileGroupsCreationTime=54, StoreTimeTaken=1
2021-02-27T04:27:24.501Z INFO hive-hive-16 org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load partition (y=2021/m=2/d=20/h=4) =666
2021-02-27T04:27:24.526Z INFO hive-hive-18 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of log files scanned => 5
2021-02-27T04:27:24.527Z INFO hive-hive-18 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner MaxMemoryInBytes allowed for compaction => 0
2021-02-27T04:27:24.527Z INFO hive-hive-18 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of entries in MemoryBasedMap in ExternalSpillableMap => 6
2021-02-27T04:27:24.527Z INFO hive-hive-18 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 4752
2021-02-27T04:27:24.527Z INFO hive-hive-18 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of entries in DiskBasedMap in ExternalSpillableMap => 0
2021-02-27T04:27:24.527Z INFO hive-hive-18 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Size of file spilled to disk => 0
2021-02-27T04:27:24.531Z INFO hive-hive-18 org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata log files from [s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093] at instant 20210227032808(dataset instant=20210227032808, metadata instant=20210227032808)
2021-02-27T04:27:24.551Z INFO hive-hive-18 org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata read for key y=2021/m=2/d=20/h=2 took [open, baseFileRead, logMerge] [537, 19, 0] ms
2021-02-27T04:27:24.560Z INFO hive-hive-18 org.apache.hudi.metadata.BaseTableMetadata Listed file in partition from metadata: partition=y=2021/m=2/d=20/h=2, #files=691
2021-02-27T04:27:24.560Z INFO hive-hive-18 org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in partition (y=2021/m=2/d=20/h=2) =691, Time taken =566
2021-02-27T04:27:24.563Z INFO hive-hive-19 org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding file-groups for partition :y=2021/m=2/d=20/h=3, #FileGroups=699
2021-02-27T04:27:24.563Z INFO hive-hive-19 org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView: NumFiles=699, NumFileGroups=699, FileGroupsCreationTime=113, StoreTimeTaken=0
2021-02-27T04:27:24.563Z INFO hive-hive-19 org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load partition (y=2021/m=2/d=20/h=3) =713
2021-02-27T04:27:24.584Z INFO hive-hive-17 org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding file-groups for partition :y=2021/m=2/d=20/h=1, #FileGroups=647
2021-02-27T04:27:24.584Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView: NumFiles=647, NumFileGroups=647, FileGroupsCreationTime=92, StoreTimeTaken=1
2021-02-27T04:27:24.584Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load partition (y=2021/m=2/d=20/h=1) =714
2021-02-27T04:27:24.609Z INFO hive-hive-19 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Total paths to process after hoodie filter 699
2021-02-27T04:27:24.615Z INFO hive-hive-16 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Total paths to process after hoodie filter 761
2021-02-27T04:27:24.631Z INFO hive-hive-17 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Total paths to process after hoodie filter 647
2021-02-27T04:27:24.670Z INFO hive-hive-18 org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding file-groups for partition :y=2021/m=2/d=20/h=2, #FileGroups=691
2021-02-27T04:27:24.670Z INFO hive-hive-18 org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView: NumFiles=691, NumFileGroups=691, FileGroupsCreationTime=110, StoreTimeTaken=0
2021-02-27T04:27:24.671Z INFO hive-hive-18 org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load partition (y=2021/m=2/d=20/h=2) =798
2021-02-27T04:27:24.714Z INFO hive-hive-18 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Total paths to process after hoodie filter 691
2021-02-27T04:27:45.364Z INFO hive-hive-17 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Reading hoodie metadata from path s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:45.364Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:45.365Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:45.410Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:45.426Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:45.427Z INFO hive-hive-17 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Found a total of 1 groups
2021-02-27T04:27:45.427Z INFO hive-hive-17 org.apache.hudi.common.table.view.FileSystemViewManager Creating InMemory based view for basePath s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:45.453Z INFO hive-hive-17 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED], [20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED], [20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:45.453Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to read 0 instants, 0 replaced file groups
2021-02-27T04:27:45.492Z INFO hive-hive-17 org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending clustering operations
2021-02-27T04:27:45.492Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:45.493Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:45.531Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/hoodie.properties
2021-02-27T04:27:45.547Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data
2021-02-27T04:27:45.547Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:45.548Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:45.585Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:45.599Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:45.618Z INFO hive-hive-17 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[0000000000000__deltacommit__COMPLETED], [20210227020224__deltacommit__COMPLETED], [20210227021755__deltacommit__COMPLETED], [20210227024056__deltacommit__COMPLETED], [20210227030430__deltacommit__COMPLETED], [20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:45.618Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView Took 0 ms to read 0 instants, 0 replaced file groups
2021-02-27T04:27:45.652Z INFO hive-hive-17 org.apache.hudi.common.util.ClusteringUtils Found 0 files in pending clustering operations
2021-02-27T04:27:45.652Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file system view for partition (files)
2021-02-27T04:27:45.709Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in partition (files) =7, Time taken =18
2021-02-27T04:27:45.710Z INFO hive-hive-17 org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding file-groups for partition :files, #FileGroups=1
2021-02-27T04:27:45.710Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView: NumFiles=7, NumFileGroups=1, FileGroupsCreationTime=1, StoreTimeTaken=0
2021-02-27T04:27:45.711Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load partition (files) =59
2021-02-27T04:27:45.711Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView Pending Compaction instant for (FileSlice {fileGroupId=HoodieFileGroupId{partitionPath='files', fileId='7e172a24-c399-4f87-a9a7-75b3d2879e0b-0'}, baseCommitTime=0000000000000, baseFile='HoodieBaseFile{fullPath=s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile, fileLen=5696, BootstrapBaseFile=null}', logFiles='[HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=25302}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=23561}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=23243}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=21732}, HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553', fileLen=27253}]'}) is :Optional.empty
2021-02-27T04:27:45.732Z INFO hive-hive-17 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[20210227020224__commit__COMPLETED], [20210227021755__commit__COMPLETED], [20210227024056__commit__COMPLETED], [20210227030430__commit__COMPLETED], [20210227032808__commit__COMPLETED], [==>20210227035211__commit__INFLIGHT]]
2021-02-27T04:27:45.756Z INFO hive-hive-17 org.apache.hudi.common.table.timeline.HoodieActiveTimeline Loaded instants [[0000000000000__deltacommit__COMPLETED], [20210227020224__deltacommit__COMPLETED], [20210227021755__deltacommit__COMPLETED], [20210227024056__deltacommit__COMPLETED], [20210227030430__deltacommit__COMPLETED], [20210227032808__deltacommit__COMPLETED]]
2021-02-27T04:27:45.756Z INFO hive-hive-17 org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata table is disabled.
2021-02-27T04:27:45.756Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView Building file system view for partition (y=2021/m=2/d=20/h=0)
2021-02-27T04:27:45.899Z INFO hive-hive-17 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:45.899Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:45.953Z INFO hive-hive-17 org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata base file from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0-15-2332_0000000000000.hfile at instant 0000000000000
2021-02-27T04:27:45.954Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Loading HoodieTableMetaClient from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:45.954Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [hdfs://hdfs-service], Config:[Configuration: ], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7dabbda3]
2021-02-27T04:27:46.012Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableConfig Loading table properties from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/.hoodie/hoodie.properties
2021-02-27T04:27:46.030Z INFO hive-hive-17 org.apache.hudi.common.table.HoodieTableMetaClient Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata
2021-02-27T04:27:46.035Z INFO hive-hive-17 org.apache.hudi.common.util.collection.DiskBasedMap Spilling to file location /tmp/view_map/c949fce3-e000-4ef4-b7c6-17ee139de8e3 in host (100.97.190.93) with hostname (presto-coordinator-5fbd7bd6bd-bhx56)
2021-02-27T04:27:46.064Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553', fileLen=0}
2021-02-27T04:27:46.082Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553 at instant 20210227020224
2021-02-27T04:27:46.086Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=0}
2021-02-27T04:27:46.102Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183', fileLen=0}
2021-02-27T04:27:46.121Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183 at instant 20210227021755
2021-02-27T04:27:46.121Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:46.134Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:46.134Z INFO hive-hive-17 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:46.137Z INFO hive-hive-17 org.apache.hudi.common.util.collection.ExternalSpillableMap Estimated Payload size => 792
2021-02-27T04:27:46.140Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=0}
2021-02-27T04:27:46.154Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049', fileLen=0}
2021-02-27T04:27:46.168Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049 at instant 20210227024056
2021-02-27T04:27:46.168Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:46.179Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:46.180Z INFO hive-hive-17 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:46.184Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=0}
2021-02-27T04:27:46.200Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088', fileLen=0}
2021-02-27T04:27:46.214Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088 at instant 20210227030430
2021-02-27T04:27:46.214Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:46.225Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:46.226Z INFO hive-hive-17 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:46.230Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieLogFormatReader Moving to the next reader for logfile HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=0}
2021-02-27T04:27:46.245Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Scanning log file HoodieLogFile{pathStr='s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093', fileLen=0}
2021-02-27T04:27:46.264Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Reading a data block from file s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093 at instant 20210227032808
2021-02-27T04:27:46.265Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:46.276Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:46.277Z INFO hive-hive-17 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:46.279Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Merging the final data blocks
2021-02-27T04:27:46.279Z INFO hive-hive-17 org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner Number of remaining logblocks to merge 1
2021-02-27T04:27:46.291Z INFO hive-hive-17 org.apache.hudi.common.fs.FSUtils Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml], FileSystem: [org.apache.hadoop.fs.PrestoFileSystemCache$FileSystemWrapper@7f02c117]
2021-02-27T04:27:46.291Z INFO hive-hive-17 org.apache.hadoop.hbase.io.hfile.CacheConfig Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=12497808, freeSize=12174472304, maxSize=12186970112, heapSize=12497808, minSize=11577621504, minFactor=0.95, multiSize=5788810752, multiFactor=0.5, singleSize=2894405376, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2021-02-27T04:27:46.293Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of log files scanned => 5
2021-02-27T04:27:46.293Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner MaxMemoryInBytes allowed for compaction => 0
2021-02-27T04:27:46.293Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of entries in MemoryBasedMap in ExternalSpillableMap => 6
2021-02-27T04:27:46.293Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 4752
2021-02-27T04:27:46.293Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Number of entries in DiskBasedMap in ExternalSpillableMap => 0
2021-02-27T04:27:46.293Z INFO hive-hive-17 org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner Size of file spilled to disk => 0
2021-02-27T04:27:46.294Z INFO hive-hive-17 org.apache.hudi.metadata.HoodieBackedTableMetadata Opened metadata log files from [s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.1_0-45-8553, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.2_0-85-17183, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.3_0-125-26049, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.4_0-168-35088, s3a://my-test-bucket/tmp/ramesh/hudi_0_7_cl2/sample_data/.hoodie/metadata/files/.7e172a24-c399-4f87-a9a7-75b3d2879e0b-0_0000000000000.log.5_0-211-44093] at instant 20210227032808(dataset instant=20210227032808, metadata instant=20210227032808)
2021-02-27T04:27:46.309Z INFO hive-hive-17 org.apache.hudi.metadata.HoodieBackedTableMetadata Metadata read for key y=2021/m=2/d=20/h=0 took [open, baseFileRead, logMerge] [396, 15, 0] ms
2021-02-27T04:27:46.314Z INFO hive-hive-17 org.apache.hudi.metadata.BaseTableMetadata Listed file in partition from metadata: partition=y=2021/m=2/d=20/h=0, #files=623
2021-02-27T04:27:46.315Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView #files found in partition (y=2021/m=2/d=20/h=0) =623, Time taken =417
2021-02-27T04:27:46.337Z INFO hive-hive-17 org.apache.hudi.common.table.view.HoodieTableFileSystemView Adding file-groups for partition :y=2021/m=2/d=20/h=0, #FileGroups=623
2021-02-27T04:27:46.338Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView addFilesToView: NumFiles=623, NumFileGroups=623, FileGroupsCreationTime=22, StoreTimeTaken=1
2021-02-27T04:27:46.338Z INFO hive-hive-17 org.apache.hudi.common.table.view.AbstractTableFileSystemView Time to load partition (y=2021/m=2/d=20/h=0) =582
2021-02-27T04:27:46.360Z INFO hive-hive-17 org.apache.hudi.hadoop.utils.HoodieInputFormatUtils Total paths to process after hoodie filter 623
2021-02-27T04:28:02.931Z DEBUG query-execution-16 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.2 is SCHEDULED
2021-02-27T04:28:02.934Z DEBUG query-execution-16 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.2 is RUNNING
2021-02-27T04:28:03.016Z DEBUG query-execution-16 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.2 is FINISHED
2021-02-27T04:28:03.024Z DEBUG query-execution-16 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.1 is FINISHED
2021-02-27T04:28:03.026Z DEBUG query-execution-16 io.prestosql.execution.StageStateMachine Stage 20210227_042722_00016_9dket.0 is FINISHED
2021-02-27T04:28:03.026Z DEBUG dispatcher-query-26 io.prestosql.execution.QueryStateMachine Query 20210227_042722_00016_9dket is FINISHING
2021-02-27T04:28:03.027Z DEBUG dispatcher-query-26 io.prestosql.execution.QueryStateMachine Query 20210227_042722_00016_9dket is FINISHED
2021-02-27T04:28:03.053Z INFO dispatcher-query-26 io.prestosql.event.QueryMonitor TIMELINE: Query 20210227_042722_00016_9dket :: Transaction:[13b23144-6f70-42e6-b876-4a69ca8744f3] :: elapsed 39914ms :: planning 179ms :: waiting 80ms :: scheduling 1724ms :: running 37991ms :: finishing 20ms :: begin 2021-02-27T04:27:23.112Z :: end 2021-02-27T04:28:03.026Z
Issue Analytics
- State:
- Created 3 years ago
- Reactions:1
- Comments:19 (8 by maintainers)
@rshanmugam1 see https://prestodb.io/blog/2020/08/04/prestodb-and-hudi and https://github.com/prestodb/presto/commit/9fd2459d98efd0809023b175ba53775466b74cc6 and https://github.com/prestodb/presto/commit/cfb2e7aa077954a02c048e81c97a47994d329852
@njalan not trino (aka prestosql), it only in prestodb