question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[SUPPORT] Spark SQL to query metadata such as hudi table snapshot

See original GitHub issue

Support Spark SQL to query metadata such as hudi table snapshot。Example:

#Iceberg sql 
SELECT * FROM bigdata.ice_arctic_demo.snapshots;
SELECT * FROM bigdata.ice_arctic_demo.history;
SELECT * FROM bigdata.ice_arctic_demo.files;
SELECT * FROM bigdata.ice_arctic_demo.manifests;

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:5 (5 by maintainers)

github_iconTop GitHub Comments

3reactions
XuQianJin-Starscommented, Mar 15, 2022

As far as I know, we couldn’t get meta info through sparkSQL for now. Fortunately, we can get these info through hoodie built-in cli. Linked https://hudi.apache.org/docs/cli

@melin @zhangyue19921010 I will use the call procedure command to obtain meta information through SparkSQL. If Directly abstract it into a table to query, and I will evaluate the change scope and workload.

0reactions
xushiyancommented, Mar 15, 2022

https://issues.apache.org/jira/browse/HUDI-3511 This jira for call procedure command to obtain meta information through SparkSQL.

thanks!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Querying Data - Apache Hudi
Once the Hudi tables have been registered to the Hive metastore, they can be queried using the Spark-Hive integration. By default, Spark SQL...
Read more >
Work with a Hudi dataset - Amazon EMR - AWS Documentation
Hudi supports inserting, updating, and deleting data in Hudi datasets through Spark. For more information, see Writing Hudi tables in Apache Hudi ......
Read more >
Table file formats - reading path: Apache Hudi
Copy-On-Write table with snapshot or read-optimized query, Merge-On-Read table with read-optimized query. Here Apache Hudi creates a ...
Read more >
[GitHub] [hudi] codejoyan commented on issue #3581: [SUPPORT ...
hudi.apache.org For queries about this service, please contact ... DataSourceReadOptions scala> spark.sql("SET hoodie.metadata.enable=true") res0: ...
Read more >
Building a Large-scale Transactional Data Lake at Uber Using ...
To this end, Hudi supports multiple query engine integrations such as Presto, Apache Hive, Apache Spark, and Apache Impala. Figure 1.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found