question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Inaccurate schema error when same exists in primary Metastore

See original GitHub issue

Assume there is a schema in primary metastore: pm_mydb

In different hive metastore, there is schema named as mydb_another.

When configuring waggle dance, we have provided prefix for external metastore as pm_.

When we ran show databases, we get following result

hive> show databases;
pm_mydb
pm_mydb_another

After this when we tried to execute query against the pm_mydb, schema within primary MS, I am getting error

hive> use  pm_mydb;

ERROR: FAILED: SemanticException [Error 10072]: Database does not exist: pm_mydb
...

When I changed the prefix to pm2_ for external MS, everything works fine.

hive> show databases;
pm_mydb
pm2_mydb_another

hive> use  pm_mydb;
LIST_OF_TABLES

It looks like that waggle dance is resolving pm_mydb as external schema (mydb schema under the MS with pm_ as prefix).

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
patduincommented, Sep 18, 2018

This is expected behaviour. Waggle Dance uses the prefix to redirect to the external metastore. The pm_ prefix is configured to go to the federated metastore. Which is in turn tell you that pm_mydb (or mydb in the external MS) doesn’t exist. This is a weird situation I suspect because the database was created bypassing Waggle Dance. Best to pick a prefix that doesn’t clash with existing databases. (or rename your database)

If this is not an option there is the alternative to switch to MANUAL database resolution mode (README has more details). This doesn’t use prefixes.

0reactions
ambrish29commented, Sep 28, 2018

@massdosage PR looks good to me.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Trying to reinstall Hive but have metastore errors
Hi, I am trying to reinstall Hive because i have issue with the metastore. I don't have an ibdata file anymore. I have...
Read more >
How to troubleshoot several Apache Hive metastore problems
Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCreateTables" org.datanucleus.store.rdbms.exceptions.
Read more >
Upgrade Amazon EMR Hive Metastore from 5.X to 6.X
To get the Metastore schema version in the current EMR 5.X cluster, run the following command in the primary node:.
Read more >
FUNCTION 'NUCLEUS_ASCII' already exists. (state=X0Y68 ...
Solution : Go to below directory: C:\hadoop_new\apache-hive-3.1.2\scripts\metastore\upgrade\derby\hive-schema-3.1.0.derby.sql comment the ...
Read more >
Troubleshooting Guide - Apache Software Foundation
This error generally occurs when the schema has evolved in backwards incompatible way by deleting some column 'col1' and we are trying to...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found