Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
hudi supports custom catalog name, spark_catalog is not mandatory
See original GitHub issueval spark = SparkSession.builder().master("local").enableHiveSupport()
.config("spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions," +
"org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
.config("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")
.config("spark.sql.catalog.spark_catalog.type", "hive")
.config("spark.sql.catalog.hudi", "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
.getOrCreate()
iceberg catalog name use spark_catalog，hudi catalog name cannot use spark_catalog。Can hudi use another name?
Hudi For Spark 3.2, the additional spark_catalog config is required: --conf ‘spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog’
Issue Analytics
- State:
- Created a year ago
- Comments:8 (6 by maintainers)
Top Results From Across the Web
Spark Guide - Apache Hudi
This guide provides a quick peek at Hudi's capabilities using spark-shell. Using Spark datasources, we will walk through.
Read more >Hudi supports `update` operation? - Stack Overflow
I have an exception when update record with spark sql for hudi as ... 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.
Read more >Get a quick start with Apache Hudi, Apache Iceberg, and Delta ...
Custom library dependencies with EMR on EKS. By default, Hudi and Iceberg are supported by Amazon EMR as out-of-the-box features.
Read more >Release 1.7.0-SNAPSHOT Apache Kyuubi Community
For most scenarios, the superpower of corresponding engines, such as Spark, and Flink, is no longer necessary. That is, most work related to...
Read more >Hive Metastore - EMR Containers Best Practices Guides
Also engineers may be required to hold the password when it is not necessary. Request: cat > Spark-Python-in-s3-hms-jdbc.json << EOF { "name": ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@melin I think you can specify
spark_catalogtoHoodieCatalogand custom catalog for iceberg catalog for a currently workaround, since Hudi currently do not support custom catalogs.@YannByron : looks like the author has given some hacky solution. Is there any enhancement we can add to hudi based on that.