Exponential backoff for retries

Hi!!!

It’s a good experience of using huey. Now…

In our usage, each individual task will make a HTTP call to another server running somewhere else. It happens that that server may be overloaded at some peak time and I would get “503 Server Error”.

I was wondering whether an exponential backoff option for retry is useful here. Since if the fixed retry window is too small, it may fail soon if the overload period of the server is long. And if the fixed retry window is too large, it may just unnecessarily wait for too much if it’s some other issue caused the retry.

[coleifer]

If you want to fiddle with task settings at run-time, you can request huey to pass the task instance into your task function, and modify it yourself.

Something like this?

@huey.task(retries=10, retry_delay=1, context=True)
def my_task(a, b, task=None):
    # task will be the task instance when the task is run by the consumer
    try:
        return do_something(a, b)
    except Exception as exc:
        task.retry_delay *= 1.15
        raise exc

[BSVogler]

Great, thank you!