question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[SUPPORT] spark-sql schema_evolution

See original GitHub issue

Describe the problem you faced

I have a hudi table

create table hudi_mor_test (
  uuid int,
  name string,
  price double,
  ts bigint
) using hudi
tblproperties (
  type = 'mor',
  primaryKey = 'uuid',
  preCombineField = 'ts'
);

then I want to change the order of the field and follow the instruction of document https://hudi.apache.org/docs/schema_evolution

alter table hudi_mor_test alter column point after uuid;

but I get a error

Error in query: ALTER COLUMN ... FIRST | ALTER is only supported with v2 tables.

I want know what is the problem

Environment Description

  • Hudi version : 0.11.1
  • Spark version : 3.2.2
  • Hive version : 2.3.9
  • Hadoop version : 2.7.3
  • Storage hdfs

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
KnightChesscommented, Aug 10, 2022

look like is hive side check, hive.metastore.disallow.incompatible.col.type.changes should be disable in you metastore

1reaction
KnightChesscommented, Aug 10, 2022

@MihawkZoro The document seems to be a little problem, try set hoodie.schema.on.read.enable=true

Read more comments on GitHub >

github_iconTop Results From Across the Web

Schema Evolution - Apache Hudi
Schema evolution allows users to easily change the current schema of a Hudi table to adapt to the data that is changing over...
Read more >
Data processing with Spark: schema evolution - Dev Genius
Managing schema evolution adds complexity in a data management system. Irrespective of the load mode, schema evolution is important for the data ...
Read more >
Advanced schema management for Spark applications at scale
Schema evolution management ... As new tables and views are created, and as table schemas evolve, schemas of views dependent on them also...
Read more >
Schema Merging (Evolution) with Parquet in Spark and Hive
With schema evolution, one set of data can be stored in multiple files with different but compatible schema. In Spark, Parquet data source...
Read more >
Schema evolution in parquet format - apache spark
Schema evolution can be (very) expensive. In order to figure out schema, you basically have to read all of your parquet files and...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found