Expiry of old results

I’ve added a small extra feature that recalculates if a result is older than a certain age.

@memory.cache(expires_after=60)
def f(x):
 ...

Will recompute if cached result is older than 60 seconds. Code is in this branch: https://github.com/fredludlow/joblib/tree/expires-after

Is this of interest? And if so, is there anything else that needs doing before sending a pull request? (I’ve added test coverage in test_memory.py)

[ShivKJ]

Is this feature in the pipline for release?

[fredludlow]

Totally agree with the philosophy of keeping it as unsurprising as possible - this wouldn’t break any existing code and the default would be to behave exactly as before.

My use case is a web app which aggregates data from a bunch of components (SQL queries and remote web APIs) and returns a bundle of data to the client. A non-cached request for this project takes maybe 10 seconds. If you use an http cache (requests_cache) that reduces to 5 seconds.If I put joblib caching around each component (there are about 10) then that reduces to miliseconds for second and subsequent calls.

I’m happy with serving up results that are, at most, a week old (the quickest changing data sources I’m using update on that timescale) but not happy with caching “forever” (the current joblib model). Without this modification my options are:

1. Set-up a cron job to clear caches every week - this breaks portability of the app (adds another installation/uninstallation step).

2. Build the cache maintenance into the request cycle either in a very broad-brushed way (delete the whole lot if root cache_dir > max_age) or per result (what this change does).

Joblib has been great for this during development because small changes to one component don’t cause a complete recalculation of the whole bundle, so the “Page Broken” --> “Make a change” --> “Hit F5” --> “Wait…” cycle is minimized.

Of course it’s your decision, if it’s not right for joblib then close this issue. Also if you think I’m going about this wrong and there’s a better way of doing it I’d be very happy to hear it - I couldn’t see any other similar library that provides this functionality, though maybe that’s because it’s a silly way to solve the problem!)