DB migration 937d04c16b64 -> 7f2635b51f5d fails

Superset upgrade from 0.29.0rc7 to 0.34.0 fails on superset db upgrade on migration mentioned above

Expected results

Success superset db upgrade

Actual results

Loaded your LOCAL configuration at [/usr/local/bin/superset_config.py]
2019-09-25 22:34:18,870:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 937d04c16b64 -> 7f2635b51f5d, update base columns
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: table _alembic_tmp_columns already exists

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/superset", line 31, in <module>
    cli()
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_migrate/cli.py", line 134, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/usr/local/lib/python3.6/site-packages/flask_migrate/__init__.py", line 95, in wrapped
    f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flask_migrate/__init__.py", line 280, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/usr/local/lib/python3.6/site-packages/alembic/command.py", line 276, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.6/site-packages/alembic/script/base.py", line 475, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.6/site-packages/alembic/util/pyfiles.py", line 90, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.6/site-packages/alembic/util/compat.py", line 156, in load_module_py
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.6/site-packages/superset/migrations/env.py", line 119, in <module>
    run_migrations_online()
  File "/usr/local/lib/python3.6/site-packages/superset/migrations/env.py", line 111, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.6/site-packages/alembic/runtime/environment.py", line 839, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.6/site-packages/alembic/runtime/migration.py", line 361, in run_migrations
    step.migration_fn(**kw)
  File "/usr/local/lib/python3.6/site-packages/superset/migrations/versions/7f2635b51f5d_update_base_columns.py", line 73, in upgrade
    batch_op.alter_column("column_name", existing_type=String(255), nullable=False)
  File "/usr/local/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.6/site-packages/alembic/operations/base.py", line 325, in batch_alter_table
    impl.flush()
  File "/usr/local/lib/python3.6/site-packages/alembic/operations/batch.py", line 102, in flush
    batch_impl._create(self.impl)
  File "/usr/local/lib/python3.6/site-packages/alembic/operations/batch.py", line 281, in _create
    op_impl.create_table(self.new_table)
  File "/usr/local/lib/python3.6/site-packages/alembic/ddl/impl.py", line 251, in create_table
    self._exec(schema.CreateTable(table))
  File "/usr/local/lib/python3.6/site-packages/alembic/ddl/impl.py", line 134, in _exec
    return conn.execute(construct, *multiparams, **params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1050, in _execute_ddl
    compiled,
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1253, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1473, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) table _alembic_tmp_columns already exists
[SQL: 
CREATE TABLE _alembic_tmp_columns (
	created_on DATETIME NOT NULL, 
	changed_on DATETIME NOT NULL, 
	id INTEGER NOT NULL, 
	column_name VARCHAR(255) NOT NULL, 
	is_active BOOLEAN, 
	type VARCHAR(32), 
	groupby BOOLEAN, 
	filterable BOOLEAN, 
	description TEXT, 
	created_by_fk INTEGER, 
	changed_by_fk INTEGER, 
	dimension_spec_json TEXT, 
	verbose_name VARCHAR(1024), 
	datasource_id INTEGER, 
	PRIMARY KEY (id), 
	CHECK (is_active IN (0, 1)), 
	CHECK (groupby IN (0, 1)), 
	CHECK (filterable IN (0, 1)), 
	CONSTRAINT fk_columns_datasource_id_datasources FOREIGN KEY(datasource_id) REFERENCES datasources (id), 
	CONSTRAINT uq_columns_column_name UNIQUE (column_name, datasource_id), 
	CONSTRAINT fk_columns_created_by_fk_ab_user FOREIGN KEY(created_by_fk) REFERENCES ab_user (id), 
	CONSTRAINT fk_columns_changed_by_fk_ab_user FOREIGN KEY(changed_by_fk) REFERENCES ab_user (id)
)

]
(Background on this error at: http://sqlalche.me/e/e3q8)

Screenshots

If applicable, add screenshots to help explain your problem.

How to reproduce the bug

1. Upgrade non-empty superset v0.29.0rc7 to 0.34.0
2. Run superset db upgrade

Environment

(please complete the following information):

• superset version: 0.34.0
• python version: 3.6.8
• node.js version: none
• npm version: none

Checklist

• I have checked the superset logs for python stacktraces and included it here as text if there are any.
• I have reproduced the issue with at least the latest released version of superset.
• I have checked the issue tracker for the same issue and I haven’t found one similar.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label #bug to this issue, with a confidence of 0.97. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[dpgaspar]

Hi, you probably have already reached this conclusion, but seems that table_columns.columns_name contains nulls, fix that first, then rerun the migration