migration plan " New dataset models " can't execute properly

A clear and concise description of what the bug is.

How to reproduce the bug

1. add new dataset using virtual query with bigquery
2. the sql is as below

SELECT
  project_id,
  user_email,
  creation_time,
  total_bytes_billed,
  query
FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE job_type = "QUERY"

1. go to console call superset db upgrade

Expected results

success

Actual results

Traceback (most recent call last): File “/usr/local/lib/python3.8/site-packages/pybigquery/sqlalchemy_bigquery.py”, line 839, in _get_table table = client.get_table(table_ref) File “/usr/local/lib/python3.8/site-packages/google/cloud/bigquery/client.py”, line 1008, in get_table api_response = self._call_api( File “/usr/local/lib/python3.8/site-packages/google/cloud/bigquery/client.py”, line 756, in _call_api return call() File “/usr/local/lib/python3.8/site-packages/google/api_core/retry.py”, line 283, in retry_wrapped_func return retry_target( File “/usr/local/lib/python3.8/site-packages/google/api_core/retry.py”, line 190, in retry_target return target() File “/usr/local/lib/python3.8/site-packages/google/cloud/_http/init.py”, line 480, in api_request raise exceptions.from_http_response(response) google.api_core.exceptions.NotFound: 404 GET https://bigquery.googleapis.com/bigquery/v2/projects/xxx_projectId/datasets/INFORMATION_SCHEMA/tables/JOBS_BY_PROJECT?prettyPrint=false: Not found: Table xxx_projectId:INFORMATION_SCHEMA.JOBS_BY_PROJECT

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File “/usr/local/bin/superset”, line 33, in <module> sys.exit(load_entry_point(‘apache-superset’, ‘console_scripts’, ‘superset’)()) File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 1128, in call return self.main(*args, **kwargs) File “/usr/local/lib/python3.8/site-packages/flask/cli.py”, line 601, in main return super().main(*args, **kwargs) File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 1053, in main rv = self.invoke(ctx) File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 754, in invoke return __callback(*args, **kwargs) File “/usr/local/lib/python3.8/site-packages/click/decorators.py”, line 26, in new_func return f(get_current_context(), *args, **kwargs) File “/usr/local/lib/python3.8/site-packages/flask/cli.py”, line 445, in decorator return __ctx.invoke(f, *args, **kwargs) File “/usr/local/lib/python3.8/site-packages/click/core.py”, line 754, in invoke return __callback(*args, **kwargs) File “/usr/local/lib/python3.8/site-packages/flask_migrate/cli.py”, line 149, in upgrade _upgrade(directory, revision, sql, tag, x_arg) File “/usr/local/lib/python3.8/site-packages/flask_migrate/init.py”, line 98, in wrapped f(*args, **kwargs) File “/usr/local/lib/python3.8/site-packages/flask_migrate/init.py”, line 185, in upgrade command.upgrade(config, revision, sql=sql, tag=tag) File “/usr/local/lib/python3.8/site-packages/alembic/command.py”, line 294, in upgrade script.run_env() File “/usr/local/lib/python3.8/site-packages/alembic/script/base.py”, line 490, in run_env util.load_python_file(self.dir, “env.py”) File “/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py”, line 97, in load_python_file module = load_module_py(module_id, path) File “/usr/local/lib/python3.8/site-packages/alembic/util/compat.py”, line 184, in load_module_py spec.loader.exec_module(module) File “<frozen importlib._bootstrap_external>”, line 843, in exec_module File “<frozen importlib._bootstrap>”, line 219, in _call_with_frames_removed File “/app/superset/extensions/…/migrations/env.py”, line 126, in <module> run_migrations_online() File “/app/superset/extensions/…/migrations/env.py”, line 118, in run_migrations_online context.run_migrations() File “<string>”, line 8, in run_migrations File “/usr/local/lib/python3.8/site-packages/alembic/runtime/environment.py”, line 813, in run_migrations self.get_context().run_migrations(**kw) File “/usr/local/lib/python3.8/site-packages/alembic/runtime/migration.py”, line 561, in run_migrations step.migration_fn(**kw) File “/app/superset/migrations/versions/b8d3a24d9131_new_dataset_models.py”, line 624, in upgrade after_insert(target=dataset) File “/app/superset/migrations/versions/b8d3a24d9131_new_dataset_models.py”, line 422, in after_insert tables = load_or_create_tables( File “/app/superset/migrations/versions/b8d3a24d9131_new_dataset_models.py”, line 282, in load_or_create_tables column_metadata = inspector.get_columns(table.table, schema=table.schema) File “/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py”, line 390, in get_columns col_defs = self.dialect.get_columns( File “/usr/local/lib/python3.8/site-packages/pybigquery/sqlalchemy_bigquery.py”, line 872, in get_columns table = self._get_table(connection, table_name, schema) File “/usr/local/lib/python3.8/site-packages/pybigquery/sqlalchemy_bigquery.py”, line 841, in _get_table raise NoSuchTableError(table_name) sqlalchemy.exc.NoSuchTableError: JOBS_BY_PROJECT

Screenshots

If applicable, add screenshots to help explain your problem.

Environment

(please complete the following information):

current superset version is helm install lastest version, I think is 1.4.2-rc1

• browser type and version:
• superset version: superset version
• python version: python --version
• node.js version: node -v
• any feature flags active:

Checklist

Make sure to follow these steps before submitting your issue - thank you!

• [ y] I have checked the superset logs for python stacktraces and included it here as text if there are any.
• [ y] I have reproduced the issue with at least the latest released version of superset.
• [ y] I have checked the issue tracker for the same issue and I haven’t found one similar.

Additional context

Add any other context about the problem here.

[jayakrishnankk]

Same problem for us.

We were testing release version 1.5

INFO  [alembic.runtime.migration] Running upgrade 5afbb1a5849b -> b8d3a24d9131, New dataset models
Traceback (most recent call last):
  File "/usr/local/bin/superset", line 33, in <module>
    sys.exit(load_entry_point('apache-superset', 'console_scripts', 'superset')())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask_migrate/cli.py", line 149, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/usr/local/lib/python3.8/site-packages/flask_migrate/__init__.py", line 98, in wrapped
    f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/flask_migrate/__init__.py", line 185, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/usr/local/lib/python3.8/site-packages/alembic/command.py", line 294, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 490, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.8/site-packages/alembic/util/compat.py", line 184, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/superset/extensions/../migrations/env.py", line 126, in <module>
    run_migrations_online()
  File "/app/superset/extensions/../migrations/env.py", line 118, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.8/site-packages/alembic/runtime/environment.py", line 813, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.8/site-packages/alembic/runtime/migration.py", line 561, in run_migrations
    step.migration_fn(**kw)
  File "/app/superset/migrations/versions/b8d3a24d9131_new_dataset_models.py", line 632, in upgrade
    after_insert(target=dataset)
  File "/app/superset/migrations/versions/b8d3a24d9131_new_dataset_models.py", line 430, in after_insert
    tables = load_or_create_tables(
  File "/app/superset/migrations/versions/b8d3a24d9131_new_dataset_models.py", line 291, in load_or_create_tables
    column_metadata = inspector.get_columns(table.table, schema=table.schema)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 390, in get_columns
    col_defs = self.dialect.get_columns(
  File "<string>", line 2, in get_columns
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 52, in cache
    ret = fn(self, con, *args, **kw)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/base.py", line 3037, in get_columns
    table_oid = self.get_table_oid(
  File "<string>", line 2, in get_table_oid
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 52, in cache
    ret = fn(self, con, *args, **kw)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/base.py", line 2952, in get_table_oid
    raise exc.NoSuchTableError(table_name)
sqlalchemy.exc.NoSuchTableError: get_periodfacts

Here get_periodfacts is a function in the postgres database, used in a dataset definition in superset datasets.

@amitmiran137 , @villebro

[ktmud]

@jayakrishnankk This migration should have been disabled in the final version of v1.5.0: https://github.com/apache/superset/blob/1.5.0/superset/migrations/versions/b8d3a24d9131_new_dataset_models.py

Can you try again?