Get Schema Registry settings from Spark config

Currently user has to provide Schema Registry settings for every invocation of a transformation function, for example:

dataFrame.select(from_avro(col("value"), schemaRegistryConfig) as 'data)

A better approach could be to take these settings from the Spark configuration:

private val paramPrefix = "abris."
private def getSchemaRegistryParams: Map[String, String] = {
    spark.conf
      .getAll
      .filterKeys(_.startsWith(paramPrefix))
      .map { case (k, v) =>
        k.substring(0, paramPrefix.length) -> v
      }
  }

Then user could specify these settings once when creating an instance of SparkSession, for example:

SparkSession.builder()
    .config(SchemaManager.PARAM_SCHEMA_REGISTRY_TOPIC, "events")
    .config(SchemaManager.PARAM_SCHEMA_REGISTRY_URL, "http://schema.registry.com")
    ...
    .getOrCreate()

The above example assumes that all Abris-specific settings start with a common prefix abris. - this makes it easier to fetch all settings at once.

Benefits:

1. More concise code - no need to specify schemaRegistryConfig on every function call
2. Can use Spark-standard ways to configure: SparkSession.builder().conf(...) in the code, spark.conf configuration file, etc. Moreover, can run Abris functions from spark-shell, thanks to --conf command-line arguments.
3. Easier and safer to write unit tests. Currently each unit test must ensure, that SchemaManager.reset() is invoked. Otherwise a previous test may spoil a subsequent test. If SchemaManager takes settings from SparkSession, then tests only need to properly stop SparkSession after each test suite.

I would suggest to rework SchemaManager to a class:

class SchemaManager(implicit spark: SparkSession) {
 
  private val schemaRegistryParams: Map[String, String] = getSchemaRegistryParams

  private def getSchemaRegistryParams: Map[String, String] = {
   // Get Abris-specific settings from the 'spark' instance
  ...
  }

}

[cerveada]

The new version of Abris will allow users to connect to unlimited number of schema registries, so we want to be able to provide different configurations for each use.

I will implement the config loading, but it will take the config from Spark as a default and when you provide any config in the expression it will override the one from spark.

[cerveada]

Hello, We had a discussion with Felipe, and we decided to not support this feature even though it could be useful in some simple use cases there are several reasons why not to do it:

• Abris is designed to be used for communication with one or more schema registries that means we still have to support normal way of configuration.
• lot of config parameters does different things depending on the function used. Having them in global config could cause some unintended behaviour.
• to decide whether a column is key or value you also need the config

Overall it would simplify most basic usage of Abris, but in other cases it would do nothing or would complicate things, and that’s why we think it’s not worth it.