APScheduler 4.0 progress tracking

I’m opening this issue as an easy way to interested parties to track development progress of the next major APScheduler release (v4.0).

Terminology changes in v4.0

The old term of “Job”, as it was, is gone, replaced by the following concepts which are closer to the terminology used by Celery:

• Task definition: a uniquely named callable coupled with configuration like maximum number of instances, misfire grace time etc.
• Schedule: binds a trigger with a task definition
• Job: queued work item for an executor (binds to a task definition, and optionally a schedule)

Also, the term “executor” is now being changed to “worker”.

Notice that the terminology may still change before the final release!

Planned major changes

v4.0 is a ground-up redesign that aims to fix all the long-standing flaws found in APScheduler over the years.

Checked boxes are changes that have already been implemented.

• Async-first design, with support for asyncio and trio (via AnyIO)
• Static typing friendly (PEP 561)
• Support for serializers other than pickle
• Broader time zone support, including zoneinfo time zones (PEP 615)
• Drop support for Python < 3.7
• Calendar interval trigger
• Stateful triggers
• threshold value for AndTrigger (resolves issues with contained IntervalTrigger instances)
• The interval trigger should start right away and not after the first interval (#375)
• Persistent store sharing among multiple schedulers (arguably the most needed feature ever for APScheduler)
• Decoupling of schedulers and workers
• Schedule-level jitter support
• Context-local job metadata information
• Easy launching of tasks immediately without needing a schedule
• Failure resilience for persistent data stores (so they don’t crash the scheduler on a temporary outage)

Potential extra features I would like to have:

• Support for tags in task definitions, schedules and jobs
• Stateful jobs
• Ability to cancel jobs
• Timeouts for jobs
• “threshold” value for OrTrigger (#453)

You will notice that I have dropped a number of features from master. Some I may never add back to v4.0, even if requested, but do voice your wishes in this issue (and this issue only – I will summarily close such requests in new tickets). Others have been removed only temporarily to give me space for the redesign.

Features on the chopping block

• Twisted scheduler (may be usable through the async scheduler if AnyIO ever gets Twisted support)
• Tornado scheduler (just use the async scheduler)
• Gevent scheduler (does not play well with the new architecture)
• Qt scheduler (difficult to test/maintain)
• Redis as a data store (may not have sophisticated enough querying capabilities)
• Rethink data store (the company has gone belly up some time ago)
• Zookeeper as a data store (may not have sophisticated enough querying capabilities)

Being on the chopping block does not mean the feature will be gone forever! It may return in subsequent minor release or even before the 4.0 final release if I deem it feasible to implement on top of the new architecture.

[agronholm]

So, I’ve been refactoring the code for the next release. There are BIG changes coming:

• The sync scheduler has been turned into a wrapper on top of the async scheduler, running in its own event loop thread (as discussed above)
• The scheduler is now the only user-facing interface, meaning that workers can’t be started separately without a scheduler, but the scheduler can be configured not to process schedules (for whatever reason). This allows me to nix the synchronous worker entirely.
• The synchronous interfaces for data stores and event brokers have been eliminated and the existing synchronous implementations have been migrated to use the async interface with worker threads instead
• ~The initialization has been changed somewhat. The data store implementation is now responsible for starting up the event broker. A base class has been provided to make this more convenient.~

All in all, this has enabled me to eliminate a lot of code, further boosting my motivation to work on this project. If you’re not okay with some of these changes, please let me know. Note that I have not yet pushed any of these changes to GitHub.

[agronholm]

I expect to make an alpha release this month. It will still lack some features, but the basics should be working. The data store system has been completely overhauled so it will most likely have some glitches which need to be ironed out over a lengthy period. I wanted to get a release out last month but alas, I have so many projects that need attention that I simply didn’t have the time. Also, the database schema is not yet stable so at least during the alpha period, you would have to start from a clean slate from time to time if you plan to use persistent data stores.

On the upside, I think I have finally resolved the last kink that prevented the schedulers from working properly. I have not pushed that code yet but I expect to do so in next new few days, once I’ve had a chance to test it a little bit locally. After that, I will start rewriting the documentation and once that’s done, I expect to make the first pre-release. I don’t want to make any promises, but this month is the target for 4.0.0a1.