Use sqlalchemy Python script to test availability of database.

Environment:

• CTFd Version/Commit:

2.0.0 branch

• Operating System:

MacOS X, but building/deploying container images.

• Web Browser and Version:

N/A

What happened?

The check using mysqladmin in docker-entrypoint.sh will fail if the target system only has MySQL libs package and doesn’t have MySQL clients package.

Further, the wiki says:

My personal suggestion is MySQL or MariaDB, but PostgreSQL will work just fine.

The docker-entrypoint.sh script is hard wired to MySQL and expects MySQL clients package to be installed. Would be nice that if also want to use PostgreSQL and only provide alternate Dockerfile and not change anything else, that this would work and didn’t have to rewrite it. Similarly, if use alternate build system like S2I for OpenShift, that can reuse existing scripts unchanged.

What did you expect to happen?

Would hope database check is database agnostic and work with only database libs available.

How to reproduce your issue

In my case hit it as building from source using S2I (Source-to-Image) builder in OpenShift. The Python base images do not include MySQL clients package.

Any associated stack traces or error logs

Waiting for ctfd-mariadb to be ready
  | /opt/app-root/src/docker-entrypoint.sh: line 33: mysqladmin: command not found
  | /opt/app-root/src/docker-entrypoint.sh: line 33: mysqladmin: command not found
  | /opt/app-root/src/docker-entrypoint.sh: line 33: mysqladmin: command not found
  | /opt/app-root/src/docker-entrypoint.sh: line 33: mysqladmin: command not found
 ...

Solution

Suggest database agnostic check be used which uses small Python script and sqlalchemy.

import os
from sqlalchemy import create_engine
database = create_engine(os.getenv('DATABASE_URL'))
database.execute('select 1')

This has added benefit that also checks that required Python packages installed correctly and work before starting up the application.

It should also technically work for SQLite if someone wanted to override DATABASE_URL and use SQLite with file system path, to directory in a mounted persistent volume, rather than have database created in default location of source code directories where not possible to mounted a persistent volume. Although in SQLite case the database file would technically need to exist first.

[GrahamDumpleton]

Then can use:

function database_ready() {
    python -c "import os, sys
from sqlalchemy_utils.functions import database_exists
database_exists(os.getenv('DATABASE_URL')) or sys.exit(1)" > /dev/null 2>&1
}

The docs for database_exists() leave a bit to be desired because it isn’t a simple case of returning True or False. It can still raise an exception such as in the case of not being able to route to the database. So still have to throw away output to avoid the exception tracebacks filling the logs.

[GrahamDumpleton]

The docker-entrypoint.sh script could use:

function database_ready() {
    python -c "import os
from sqlalchemy import create_engine
database = create_engine(os.getenv('DATABASE_URL'))
database.execute('select 1')" > /dev/null 2>&1
}

# Check that the database is available
if [ -n "$DATABASE_URL" ]; then
    database=`echo $DATABASE_URL | awk -F[@//] '{print $4}'`
    echo "Waiting for $database to be ready"
    while ! database_ready; do
        # Show some progress
        echo -n '.';
        sleep 1;
    done
    echo "$database is ready"
    # Give it another second.
    sleep 1;
fi

There is another existing problem with that loop for checking the database though.

        echo -n '.';

This attempt at showing progress will not necessarily work when run in a container as display of output in log system of the container platform can be line buffered. This means you will only see the dots all at once when it finally succeeds, or possibly if some line buffering limit is reached. Would be better to use:

        echo 'wait 1 sec for database';

and accept that it will display something every second, one per line. If too noisy, increase sleep time to 2 seconds to at least halve it.