Nesting multiple decorators breaks waiting times

Look at this simple example and notice the waiting times does not increase as expected:

In [23]: def backoff_hdlr(details):
    ...:     print ("Backing off {wait:0.1f} seconds afters {tries} tries "
    ...:            "calling function {target} with args {args} and kwargs "
    ...:            "{kwargs}".format(**details))
    ...:
    ...: @backoff.on_exception(backoff.expo, ValueError,on_backoff=backoff_hdlr)
    ...: @backoff.on_exception(backoff.expo, TypeError,on_backoff=backoff_hdlr)
    ...: def get_url(url):
    ...:     raise ValueError
    ...:
    ...: get_url("")
Backing off 0.8 seconds afters 1 tries calling function <function get_url at 0x10a7e2048> with args ('',) and kwargs {}
Backing off 1.3 seconds afters 2 tries calling function <function get_url at 0x10a7e2048> with args ('',) and kwargs {}
Backing off 3.2 seconds afters 3 tries calling function <function get_url at 0x10a7e2048> with args ('',) and kwargs {}
Backing off 0.5 seconds afters 4 tries calling function <function get_url at 0x10a7e2048> with args ('',) and kwargs {}
Backing off 2.6 seconds afters 5 tries calling function <function get_url at 0x10a7e2048> with args ('',) and kwargs {}
Backing off 3.7 seconds afters 6 tries calling function <function get_url at 0x10a7e2048> with args ('',) and kwargs {}
Backing off 2.3 seconds afters 7 tries calling function <function get_url at 0x10a7e2048> with args ('',) and kwargs {}
Backing off 90.7 seconds afters 8 tries calling function <function get_url at 0x10a7e2048> with args ('',) and kwargs {}
Backing off 175.3 seconds afters 9 tries calling function <function get_url at 0x10a7e2048> with args ('',) and kwargs {}
Backing off 146.8 seconds afters 10 tries calling function <function get_url at 0x10a7e2048> with args ('',) and kwargs {}
Backing off 479.5 seconds afters 11 tries calling function <function get_url at 0x10a7e2048> with args ('',) and kwargs {}

(see the 0.5 seconds in the 4th log)

If we just leave one exponential backoff now behaves as expected:

In [25]: def backoff_hdlr(details):
    ...:     print ("Backing off {wait:0.1f} seconds afters {tries} tries "
    ...:            "calling function {target} with args {args} and kwargs "
    ...:            "{kwargs}".format(**details))
    ...:
    ...: @backoff.on_exception(backoff.expo, ValueError,on_backoff=backoff_hdlr)
    ...: def get_url(url):
    ...:     raise ValueError
    ...:
    ...: get_url("")
Backing off 0.2 seconds afters 1 tries calling function <function get_url at 0x10a32d620> with args ('',) and kwargs {}
Backing off 0.2 seconds afters 2 tries calling function <function get_url at 0x10a32d620> with args ('',) and kwargs {}
Backing off 2.1 seconds afters 3 tries calling function <function get_url at 0x10a32d620> with args ('',) and kwargs {}
Backing off 4.8 seconds afters 4 tries calling function <function get_url at 0x10a32d620> with args ('',) and kwargs {}
Backing off 10.5 seconds afters 5 tries calling function <function get_url at 0x10a32d620> with args ('',) and kwargs {}

Tested with backoff ==1.8.1 and saw no relevant changes in the changelog since then.

[bgreen-litl]

Woah, I just looked at my own at example more carefully and it’s weirder than that. The KeyError wait generator is getting reset each time. I think combining decorators doesn’t work as neatly as was supposed. 😕

[martinvol]

I think you’re right that at the least it could be documented better. At of curiosity, do you have a real world case where you wanted to define different behavior for different exceptions?

Yes, just hitting an API and if I get an exception related with bad formatting I don’t want to retry, but if I hit any other exception I want to back off exponentially.