Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[SUPPORT] Support for Confluent Cloud SchemaRegistryProvider

See original GitHub issue

Describe the problem you faced Sharing here as requested on Slack

I would like to configure a DeltaStreamer SchemaRegistryProvider to connect to a Confluent Schema Registry hosted on Confluent Cloud. This connection requires Basic HTTP authentication (to provide a Confluent Cloud API + Secret Key), which the current SchemaRegistryProvider does not support. Are there alternative ways to make this connection?

https://github.com/apache/hudi/blob/c21209cb58a5cff3267aca9f17fe7d2c4edd3190/hudi-utilities/src/main/java/org/apache/hudi/utilities/schema/SchemaRegistryProvider.java#L54-L59

Environment Description

Hudi version : 0.5.3
Spark version : 2.4.5
Hive version : 2.3.7
Hadoop version : 2.8.5
Storage (HDFS/S3/GCS…) : S3
Running on Docker? (yes/no) : no

Additional context AWS EMR

Issue Analytics

State:
Created 3 years ago
Comments:5 (2 by maintainers)

Top GitHub Comments

1reaction

tmacedocommented, Jun 9, 2021

I’m surprised this worked for you both as I was trying this and stumbled upon https://bugs.openjdk.java.net/browse/JDK-5043482

This is also on EMR, so I assume you’re not using an alternative JDK?

0reactions

bvaradarcommented, Aug 11, 2020

@jpugliesi : With Spark DataSource write the schema is implicitly derived from the input data-frame we want to write. Is there a specific use-case you have in mind ?

Since DeltaStreamer is a generic ingestion tool, it made sense to provide a framework to plugin schema providers.

Top Results From Across the Web

Confluent Cloud Support

The Developer support plan is intended for non-production use cases, and covers components that are fully managed by Confluent. Self-managed services, which are ......

Confluent Cloud Support - Managed Kafka® as a Service

Confluent Support: Get access to world class 24/7 support with Confluent Cloud's event streaming services, or share knowledge in our community forum.

Schema Registry and Confluent Cloud

The Confluent Platform installation provides the binaries for Schema Registry that run locally and connect to your Confluent Cloud cluster. You do not...

Supported Versions and Interoperability for Confluent Platform

This topic provides the supported versions and interoperability of Confluent Platform and its components. Confluent Platform and Apache Kafka Compatibility¶ ...

Manage Schemas in Confluent Cloud

Schema Management is fully supported on Confluent Cloud with the per-environment, hosted Schema Registry, and is a key element of Stream Governance on ......