question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Example for schema evolution?

See original GitHub issue

Hi there,

Looking at the docs, you reference “Compatibility and schema evolution”, but I am unable to find any example on how to achieve schema evolution, which is one of the main benefits of using Avro.

For example: I would like new consumers to be able to read old producer data. Does this library support this?

For example:

V1 schema

{
    "type": "record",
    "name": "Foo",
    "fields": [
        {
            "name": "f1",
            "type": "string"
        }
    ]
}

V2 schema

{
    "type": "record",
    "name": "Foo",
    "fields": [
        {
            "name": "f1",
            "type": "string"
        },
        {
            "name": "f2",
            "type": "string",
            "default": "",
        }
    ]
}

I would like V2 consumers to have the property f2 with value "" when reading data produced with schema v1, but this does not happen, since the message is decoded using the old schema

Issue Analytics

  • State:open
  • Created 4 years ago
  • Reactions:1
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
nick-zhcommented, Oct 7, 2021

I just saw, there is already an open PR for this #137

1reaction
ivankcommented, Jul 8, 2021

At our shop we had developed a simpler implementation of schema registry back in the day, since this project was not yet available. Its called Castle (as in "Kafka’s greatest work 😃).

Anyway we’ve recently encountered this issue ourselves, so we did implement reader schemas.

You guys are right, Avsc does support it with createResolver, and it’s pretty easy to implement. I’d be happy to give it a go and try to write a PR for it here too, if I had some guidance where you’d think its best to add it.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Schema Evolution and Compatibility - Confluent Documentation
An important aspect of data management is schema evolution. After the initial schema is defined, applications may need to evolve it over time....
Read more >
Schema Evolution
Schema evolution is the term used for how the store behaves when Avro schema is changed after data has been written to the...
Read more >
Practical Schema Evolution with Avro | by Elliot West - Medium
— Apache Avro has the notion of schema compatibility that allows us to determine whether or not a schema is compatible with one...
Read more >
Diving Into Delta Lake: Schema Enforcement & Evolution
Learn how schema enforcement and schema evolution work together on Delta Lake to ensure high quality, reliable data.
Read more >
Schema Evolution - Apache Hudi
Schema evolution is a very important aspect of data management. ... Let us walk through an example to demonstrate the schema evolution support...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found