Export from 3.4.3 causing missing content when imported into 3.5.0

Environment:

• CTFd Version/Commit: 3.5.0
• Operating System: Ubuntu
• Web Browser and Version: Vivaldi

What happened? Exported backup from 3.4.3 (using official Docker images) imported into 3.5.0 container.

When import was complete the following page stayed refreshing but with no changes. Decided to try to visit other pages after 15 minutes of this…

Then many error 500’s, challenges page empty, only three user accounts visible from around 10, system admin panel not available.

What did you expect to happen? Import should work between versions.

How to reproduce your issue Uncertain.

Any associated stack traces or error logs

Visiting the scoreboard:

Logs:

172.17.0.1 - - [20/May/2022:19:15:44 +0000] "GET /scoreboard HTTP/1.1" 200 7539 "http://127.0.0.1:8001/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.162 Safari/537.36"
ERROR [CTFd] Exception on /api/v1/scoreboard/top/10 [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1283, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: no such column: challenges.next_id

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.9/site-packages/flask_restx/api.py", line 403, in wrapper
    resp = resource(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flask/views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flask_restx/resource.py", line 49, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/opt/CTFd/CTFd/utils/decorators/visibility.py", line 92, in _check_account_visibility
    return f(*args, **kwargs)
  File "/opt/CTFd/CTFd/utils/decorators/visibility.py", line 25, in _check_score_visibility
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flask_caching/__init__.py", line 436, in decorated_function
    rv = f(*args, **kwargs)
  File "/opt/CTFd/CTFd/api/v1/scoreboard.py", line 114, in get
    "value": solve.challenge.value,
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 287, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 723, in get
    value = self.callable_(state, passive)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 749, in _load_for_state
    return self._emit_lazyload(
  File "<string>", line 1, in <lambda>
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 837, in _emit_lazyload
    q(session)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/ext/baked.py", line 616, in _load_on_pk_identity
    result = list(bq.for_session(self.session).params(**params))
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/ext/baked.py", line 445, in __iter__
    return q._execute_and_instances(context)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1133, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1323, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1517, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1283, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such column: challenges.next_id
[SQL: SELECT challenges.id AS challenges_id, challenges.name AS challenges_name, challenges.description AS challenges_description, challenges.connection_info AS challenges_connection_info, challenges.next_id AS challenges_next_id, challenges.max_attempts AS challenges_max_attempts, challenges.value AS challenges_value, challenges.category AS challenges_category, challenges.type AS challenges_type, challenges.state AS challenges_state, challenges.requirements AS challenges_requirements
FROM challenges
WHERE challenges.id = ?]
[parameters: (1,)]
(Background on this error at: http://sqlalche.me/e/e3q8)

[ColdHeat]

Importing should work fine on a MySQL/MariaDB database. The intention of the import/export system is to be able to save data from an older version and then install it on a newer version.

It isn’t intended for people making modifications to imports or custom building imports but it is doable in theory.

I feel the issue here is related mostly to sqlite because we can’t do migrations on sqlite easily. The plain Docker image bundles sqlite just b/c it’s easier. And it’s easier to put MySQL on a seperate container and use it via docker-compose.

[huornlmj]

Should the Docker Hub image (https://hub.docker.com/r/ctfd/ctfd) then not be updated to remove the built-in sqlite database?