BackgroundScheduler.get_jobs() hangs when used with Flask and Sqlalchemy

Motivation

On startup, I’d like to be able to add a persistent job store and add a job only if that job store is empty.

What works

With sqlalchemy, but without flask, the following code works:

from time import sleep
import sqlalchemy as sa

# Connect to example.sqlite and add a new job only if there are no jobs already.

from apscheduler.schedulers.background import BackgroundScheduler

log = print

engine = sa.create_engine('sqlite:///{}'.format('example.sqlite'))

def alarm():
    print('Alarm')

if __name__ == '__main__':
    scheduler = BackgroundScheduler()
    log("created scheduler")

    scheduler.add_jobstore('sqlalchemy', engine=engine)
    log("Added jobstore")

    scheduler.start()
    log("Started scheduler")
    if not scheduler.get_jobs():
        log("Added job")
        scheduler.add_job(alarm, 'interval', seconds=20)
    else:
        log("Didn't add job.")
    try:
        while True:
            sleep(2)
    except (KeyboardInterrupt, SystemExit):
        pass

With Flask but without sqlalchemy, the following works. It doesn’t make use of persistent storage, of course, but get_jobs() will return [].:

import os

from time import sleep
import flask

from apscheduler.schedulers.background import BackgroundScheduler

# Verify that apscheduler works with flask, as long as we don't use
# persistent storage.

# run with
# & { $env:FLASK_APP='demo.py'; $env:FLASK_DEBUG=1; python -m flask
 run}

app = flask.Flask(__name__)
log = app.logger.info
log("Created App")


def alarm():
    print('Alarm')


if not app.debug or os.environ.get("WERKZEUG_RUN_MAIN") == 'true':
    scheduler = BackgroundScheduler()
    log("created scheduler")

    scheduler.start()
    log("Started scheduler")
    if not scheduler.get_jobs():
        app.logger.info("Added job")
        scheduler.add_job(alarm, 'interval', seconds=20)
    else:
        app.logger.info("Didn't add job.")

What doesn’t work:

When I try to add a persistent job store to this flask app, scheduler.get_jobs() hangs:

import os

from time import sleep
import sqlalchemy as sa
import flask

from apscheduler.schedulers.background import BackgroundScheduler

# Hangs at get_jobs()

app = flask.Flask(__name__)
log = app.logger.info
log("Created App")

### NEW ###
engine = sa.create_engine('sqlite:///{}'.format('example.sqlite'))
###########

def alarm():
    print('Alarm')

# Don't create two schedulers when running in debug mode.
if not app.debug or os.environ.get("WERKZEUG_RUN_MAIN") == 'true':
    scheduler = BackgroundScheduler()
    log("created scheduler")

    ### NEW ###
    scheduler.add_jobstore('sqlalchemy', engine=engine)
    log("Added jobstore")
    ###########

    scheduler.start()
    log("Started scheduler")
    if not scheduler.get_jobs():
        app.logger.info("Added job")
        scheduler.add_job(alarm, 'interval', seconds=20)
    else:
        app.logger.info("Didn't add job.")

Environment

Windows 10 Python 3.5.3

Running in a virtual environment with:

Package      Version
------------ -------
APScheduler  3.4.0
click        6.7
Flask        0.12.2
itsdangerous 0.24
Jinja2       2.10
MarkupSafe   1.0
pip          9.0.1
pytz         2017.3
setuptools   37.0.0
six          1.11.0
SQLAlchemy   1.1.15
tzlocal      1.4
Werkzeug     0.12.2
wheel        0.30.0

Edit:

This also appears on Linux with 3.6.2. All the package versions are the same.

[btalberg]

Thanks for responding @agronholm. I dropped flask-apscheduler and am now using APScheduler directly. I am able to replicate the issue. I’ve done some debugging and have found a deadlock caused in relation to _jobstores_lock.

After starting APScheduler a thread is spun off which executes _process_jobs, this sets the _jobstores_lock here: https://github.com/agronholm/apscheduler/blob/cbf2eeb21695343c1996e59732adbc8fbbab6842/apscheduler/schedulers/base.py#L929

A query executed within the context of this lock by a function called get_due_jobs never returns. I followed the execution down to the last executed line and it seems there is an issue receiving a new cursor from SQLAlchemy’s connection pool (https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/pool.py#L970).

Simultaneously, while the ^^^ request hangs, the main thread executes add_job which also requests a lock from _jobstores_lock. That lock request occurs here: https://github.com/agronholm/apscheduler/blob/cbf2eeb21695343c1996e59732adbc8fbbab6842/apscheduler/schedulers/base.py#L428

My config

init.py

from apscheduler.schedulers.background import BackgroundScheduler
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate

app = Flask(__name__)

app.config.from_object('config.default')

db = SQLAlchemy(app)
migrate = Migrate(app, db)

scheduler = BackgroundScheduler()
scheduler.add_jobstore('sqlalchemy', engine=db.engine)
scheduler.start()

def job():
    print("HELLO JOB")

app.logger.info("Adding job")
scheduler.add_job(job, 'interval', minutes=10, replace_existing=True, id='test_job')
app.logger.info("Added job")

config/default.py

DEBUG=True
SQLALCHEMY_TRACK_MODIFICATIONS=False
SECRET_KEY="SOMETHINGSECRET"
MAIL_SUPPRESS_SEND=True
SQLALCHEMY_DATABASE_URI="postgresql://wf@localhost:5432/wf_development"
SQLALCHEMY_ECHO=True
PROPAGATE_EXCEPTIONS=True

Output

 * Serving Flask app "app"
 * Forcing debug mode on
2017-11-29 13:33:49,194 INFO sqlalchemy.engine.base.Engine select version()
2017-11-29 13:33:49,195 INFO sqlalchemy.engine.base.Engine {}
2017-11-29 13:33:49,197 INFO sqlalchemy.engine.base.Engine select current_schema()
2017-11-29 13:33:49,197 INFO sqlalchemy.engine.base.Engine {}
2017-11-29 13:33:49,198 INFO sqlalchemy.engine.base.Engine SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1
2017-11-29 13:33:49,198 INFO sqlalchemy.engine.base.Engine {}
2017-11-29 13:33:49,199 INFO sqlalchemy.engine.base.Engine SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1
2017-11-29 13:33:49,199 INFO sqlalchemy.engine.base.Engine {}
2017-11-29 13:33:49,200 INFO sqlalchemy.engine.base.Engine show standard_conforming_strings
2017-11-29 13:33:49,200 INFO sqlalchemy.engine.base.Engine {}
2017-11-29 13:33:49,201 INFO sqlalchemy.engine.base.Engine select relname from pg_class c join pg_namespace n on n.oid=c.relnamespace where pg_catalog.pg_table_is_visible(c.oid) and relname=%(name)s
2017-11-29 13:33:49,201 INFO sqlalchemy.engine.base.Engine {'name': u'apscheduler_jobs'}
--------------------------------------------------------------------------------
INFO in __init__ [./app/__init__.py:20]:
Adding job
--------------------------------------------------------------------------------

[tmeumann-ais]

Yes this is still relevant. I ran into this issue a few days ago using gunicorn, flask, apscheduler and sqlalchemy.