Internal Server Error when importing juiceshop challenges due to lack of SQLite migration support

Environment:

• CTFd Version/Commit: 3.4.0 / https://github.com/CTFd/CTFd/commit/d03692d002938df0eef43404d38035e53c134fe2
• Operating System: Unix 4.15.0.135
• Web Browser and Version: Edge 93.0.961.47

What happened? I’m trying to import the zip file that I exported from my configured juiceshop instance. This didn’t work, I’m getting a database related error which states:

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such column: pages.format

What did you expect to happen? I expected the import to work.

How to reproduce your issue I used the latest versions of both juiceshop as well as CTFd. I’ve exported the juiceshop challenges using the juice-shop-ctf command, which produces the zip file. I then created a new instance of CTFd and imported the zip file, which triggers the error for me. I didn’t do any customization to either.

Any associated stack traces or error logs

2021-09-17T20:19:59.275871494Z ERROR [CTFd] Exception on /setup [GET]
2021-09-17T20:19:59.275909194Z Traceback (most recent call last):
2021-09-17T20:19:59.275916794Z   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
2021-09-17T20:19:59.275921294Z     cursor, statement, parameters, context
2021-09-17T20:19:59.275925094Z   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
2021-09-17T20:19:59.275928894Z     cursor.execute(statement, parameters)
2021-09-17T20:19:59.275932494Z sqlite3.OperationalError: no such column: pages.format
2021-09-17T20:19:59.275936194Z
2021-09-17T20:19:59.275939494Z The above exception was the direct cause of the following exception:
2021-09-17T20:19:59.275942994Z
2021-09-17T20:19:59.275946294Z Traceback (most recent call last):
2021-09-17T20:19:59.275949794Z   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
2021-09-17T20:19:59.275953394Z     response = self.full_dispatch_request()
2021-09-17T20:19:59.275956794Z   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
2021-09-17T20:19:59.275960394Z     rv = self.handle_user_exception(e)
2021-09-17T20:19:59.275963894Z   File "/usr/local/lib/python3.7/site-packages/flask_restx/api.py", line 639, in error_router
2021-09-17T20:19:59.275967494Z     return original_handler(e)
2021-09-17T20:19:59.275970894Z   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
2021-09-17T20:19:59.275974794Z     reraise(exc_type, exc_value, tb)
2021-09-17T20:19:59.275978294Z   File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
2021-09-17T20:19:59.275981994Z     raise value
2021-09-17T20:19:59.275985394Z   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
2021-09-17T20:19:59.275988994Z     rv = self.dispatch_request()
2021-09-17T20:19:59.275992394Z   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
2021-09-17T20:19:59.275996094Z     return self.view_functions[rule.endpoint](**req.view_args)
2021-09-17T20:19:59.276009294Z   File "/opt/CTFd/CTFd/views.py", line 261, in setup
2021-09-17T20:19:59.276013394Z     return render_template("setup.html", state=serialize(generate_nonce()))
2021-09-17T20:19:59.276016994Z   File "/usr/local/lib/python3.7/site-packages/flask/templating.py", line 140, in render_template
2021-09-17T20:19:59.276020694Z     ctx.app,
2021-09-17T20:19:59.276024094Z   File "/usr/local/lib/python3.7/site-packages/flask/templating.py", lin
e 120, in _render
2021-09-17T20:19:59.276027694Z     rv = template.render(context)
2021-09-17T20:19:59.276031094Z   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
2021-09-17T20:19:59.276034794Z     self.environment.handle_exception()
2021-09-17T20:19:59.276038294Z   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
2021-09-17T20:19:59.276042994Z     reraise(*rewrite_traceback_stack(source=source))
2021-09-17T20:19:59.276046594Z   File "/usr/local/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
2021-09-17T20:19:59.276050194Z     raise value.with_traceback(tb)
2021-09-17T20:19:59.276053594Z   File "/opt/CTFd/CTFd/themes/core/templates/setup.html", line 1, in top-level template code
2021-09-17T20:19:59.276057194Z     {% extends "base.html" %}
2021-09-17T20:19:59.276060794Z   File "/opt/CTFd/CTFd/themes/core/templates/base.html", line 29, in top-level template code
2021-09-17T20:19:59.276064494Z     {% include "components/navbar.html" %}
2021-09-17T20:19:59.276067994Z   File "/opt/CTFd/CTFd/themes/core/templates/components/navbar.html", line 16, in top-level template code
2021-09-17T20:19:59.276071594Z     {% for page in Plugins.user_menu_pages %}
2021-09-17T20:19:59.276075094Z   File "/usr/local/lib/python3.7/site-packages/jinja2/sandbox.py", line 407, in getattr
2021-09-17T20:19:59.276078694Z     value = getattr(obj, attribute)
2021-09-17T20:19:59.276081994Z   File "/opt/CTFd/CTFd/constants/plugins.py", line 47, in user_menu_pages
2021-09-17T20:19:59.276097494Z     return get_user_page_menu_bar()
2021-09-17T20:19:59.276101094Z   File "/opt/CTFd/CTFd/plugins/__init__.py", line 149, in get_user_page_menu_bar
2021-09-17T20:19:59.276104794Z     for p in get_pages() + app.plugin_menu_bar:
2021-09-17T20:19:59.276108294Z   File "/usr/local/lib/python3.7/site-packages/flask_caching/__init__.py", line 834, in decorated_function
2021-09-17T20:19:59.276112194Z     rv = f(*args, **kwargs)
2021-09-17T20:19:59.276115594Z   File "/opt/CTFd/CTFd/utils/config/pages.py", line 55, in get_pages
2021-09-17T20:19:59.276119294Z     Pages.route != "index", Pages.draft.isnot(True), Pages.hidden.isnot(True)
2021-09-17T20:19:59.276122994Z   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3319, in all
2021-09-17T20:19:59.276126694Z     return list(self)
2021-09-17T20:19:59.284499706Z   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
2021-09-17T20:19:59.284511506Z     return self._execute_and_instances(context)
2021-09-17T20:19:59.284515806Z   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
2021-09-17T20:19:59.284519706Z     result = conn.execute(querycontext.statement, self._params)
2021-09-17T20:19:59.284523406Z   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
2021-09-17T20:19:59.284527306Z     return meth(self, multiparams, params)
2021-09-17T20:19:59.284530906Z   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
2021-09-17T20:19:59.284534706Z     return connection._execute_clauseelement(self, multiparams, params)
2021-09-17T20:19:59.284540106Z   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
2021-09-17T20:19:59.2845
44106Z     distilled_params,
2021-09-17T20:19:59.284556906Z   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
2021-09-17T20:19:59.284562006Z     e, statement, parameters, cursor, context
2021-09-17T20:19:59.284565706Z   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
2021-09-17T20:19:59.284569406Z     sqlalchemy_exception, with_traceback=exc_info[2], from_=e
2021-09-17T20:19:59.284573106Z   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
2021-09-17T20:19:59.284576906Z     raise exception
2021-09-17T20:19:59.284580406Z   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
2021-09-17T20:19:59.284584206Z     cursor, statement, parameters, context
2021-09-17T20:19:59.284587806Z   File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
2021-09-17T20:19:59.284591606Z     cursor.execute(statement, parameters)
2021-09-17T20:19:59.284595106Z sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such column: pages.format

I am not 100% sure what’s to blame here: the juiceshop export file or the import routine. But if the export file is wrong somehow I would expect CTFd to complain about that.

[ColdHeat]

The issue here is that you’re using SQLite instead of MySQL/MariaDB. The docs should be updated to reflect this expectation but the recommended server is MySQL/MariaDB https://docs.ctfd.io/docs/deployment/#database-server.

SQLite doesn’t really do migrations very well (last I checked) so it’s hard to import the data from Juice Shop at it’s older migration version and then upgrade. This should work in MySQL but when I have time I will investigate how this either be made to work in SQLite or suggest changes in Juice Shop CLI.

[elonmallin]

I might be wrong but I think SQLite is quite fitting for small to medium sized CTFs. It would also make CTFd extremely easy to deploy since all you need is the docker image and import/export to setup/template/share/recover CTFs.

Ex:

• Spin it up locally and setup a starting point for your company CTFs and export.
• Spin it up and import the starting point and prepare a CTF and export.
• Share the exports with anyone and they can contribute by spin up, import, change, export and share again.
• Or deploy and collaborate then export.
• When you’re ready to host, deploy the image and import.
• Backup during CTF and archive after.

It’s already quite easy I guess but setting up a DB is a bit of a hassle depending on infrastructure.

I’ve tried adding render_as_batch in env.py and some more migrate boilerplate I’ve found in different places but I’m kind of fumbling in the dark with this.

If someone has some knowledge about this stuff or hints would be nice 😃

I kinda like the idea of the above “workflow” but tbh I’ve never hosted myself and first time looking at this so not sure it’s a valid point for smaller things?