Concurrently Executed, Non-reoccurring Schedules Crashing (4.x)

Describe the bug Concurrently executed, non-reoccurring schedules appear to be crashing the scheduler with this output:

Scheduler crashed
Traceback (most recent call last):
  File "/home/runner/APScheduler-4-Crash-Date-Schedules-Non-Reoccurring/venv/lib/python3.8/site-packages/apscheduler/schedulers/async_.py", line 303, in run
    await self.data_store.release_schedules(self.identity, schedules)
  File "/home/runner/APScheduler-4-Crash-Date-Schedules-Non-Reoccurring/venv/lib/python3.8/site-packages/apscheduler/datastores/async_adapter.py", line 74, in release_schedules
    await to_thread.run_sync(self.original.release_schedules, scheduler_id, schedules)
  File "/home/runner/APScheduler-4-Crash-Date-Schedules-Non-Reoccurring/venv/lib/python3.8/site-packages/anyio/to_thread.py", line 28, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(func, *args, cancellable=cancellable,
  File "/home/runner/APScheduler-4-Crash-Date-Schedules-Non-Reoccurring/venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 818, in run_sync_in_worker_thread
    return await future
  File "/home/runner/APScheduler-4-Crash-Date-Schedules-Non-Reoccurring/venv/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 754, in run
    result = context.run(func, *args)
  File "/home/runner/APScheduler-4-Crash-Date-Schedules-Non-Reoccurring/venv/lib/python3.8/site-packages/apscheduler/datastores/memory.py", line 200, in release_schedules
    index = self._find_schedule_index(schedule_state)
  File "/home/runner/APScheduler-4-Crash-Date-Schedules-Non-Reoccurring/venv/lib/python3.8/site-packages/apscheduler/datastores/memory.py", line 91, in _find_schedule_index
    return self._schedules.index(state, left_index, right_index + 1)
ValueError: ScheduleState(schedule=Schedule(id='TASK_524df3c2-ef34-418b-9215-4436a103415d', task_id='__main__:async_task', trigger=DateTrigger('2022-04-14 21:59:51+00:00'), args=(), kwargs={'task_id': 'TASK_524df3c2-ef34-418b-9215-4436a103415d'}, coalesce=<CoalescePolicy.latest: 2>, misfire_grace_time=None, max_jitter=None, tags=frozenset(), next_fire_time=datetime.datetime(2022, 4, 14, 21, 59, 51, tzinfo=datetime.timezone.utc), last_fire_time=None, acquired_by=None, acquired_until=None), next_fire_time=datetime.datetime(2022, 4, 14, 21, 59, 51, tzinfo=datetime.timezone.utc), acquired_by='83664131ea7d-372-140004368854720', acquired_until=datetime.datetime(2022, 4, 14, 22, 0, 21, 14848, tzinfo=datetime.timezone.utc)) is not in list

To Reproduce

from uuid import uuid4
from datetime import (
  datetime as Datetime, 
  timedelta as Timedelta,
)

from anyio import run, sleep
from apscheduler.schedulers.async_ import AsyncScheduler
from apscheduler.triggers.date import DateTrigger
from apscheduler.events import Event

async def async_task(task_id):
  await sleep(1)

def listener(event: Event):
  print(event)
  
async def main():
  schedule_count = 6
  now = Datetime.utcnow()
  # Schedules should all be added before this trigger time
  trigger_datetime = (now + Timedelta(seconds=3)).replace(microsecond=0)

  # Create the async scheduler
  scheduler = AsyncScheduler()
  scheduler.events.subscribe(listener)

  # Create the concurrent schedules
  async with scheduler:
    for i in range(schedule_count):
      await scheduler.add_schedule(
        func_or_task_id=async_task,
        trigger=DateTrigger( # Single time to make these schedules non-reoccurring
          run_time=trigger_datetime,
        ),
        id=f'TASK_{uuid4()}',
      )

    # Loop forever
    while True:
      await sleep(0)

run(main)

Expected behavior The jobs for these concurrently scheduled tasks would all run successfully.

Additional context Most of the time concurrent, reoccurring schedules do not run into this issue.

For example, this schedule that runs yearly, can often be run concurrently with other same schedules without issue:

CronTrigger(
  timezone='UTC',
  year='*',
  month=trigger_datetime.month,
  day=trigger_datetime.day,
  day_of_week='*',
  hour=trigger_datetime.hour,
  minute=trigger_datetime.minute,
  second=trigger_datetime.second,
)

However, I have noticed one other edge case that produces the same crash above if you try to add additional reoccurring, concurrent schedules while existing schedules with the same timing are currently being handled/triggered by the scheduler. This case is a bit harder to observe, but can be reproduced by creating many schedules at the same time, same schedule that are all scheduled to run while you are creating them.

Thanks for all your help on this issue! Happy to provide more details if I can help.

[agronholm]

Figured out 1) too: the block that released the schedules was indented too deep, causing other schedules to be released too early. I’ve pushed a fix for that too.

[agronholm]

The quick answer to 2) is that the exception handler block is missing raise in AsyncScheduler.