FANCY_REMEMBER_ALL_URLS option and performance over time

Are there known limitations to this option? I had enabled it to give more visibility to our system and at about ~22,000 entries, the endpoints that were being tracked stopped being functional. Clearing the fancy-urls key solved the issue immediately.

In [1]: v = cache.get('fancy-urls')

In [2]: len(v.keys())
Out[2]: 22717

Above was the final count before deleting the key.

[YPCrumble]

@peterbe I’m coming up against this too due to the default Memcached limit of 1MB. I like the idea of using the separate key for each line, but like you point out, Memcached doesn’t have the search functionality for “searching” cache keys.

One option I can think of is to “shard” the fancy_cache URLs by x time period and only keep n periods of data. So if X is one day, and n is one week, we would have seven keys at any given time and would concatenate these seven into one dict that is the equivalent of the current implementation’s dict.

I’ve been testing #45 with success on my server and would love to incorporate a fix to this issue as well as I update that!

Thanks again for maintaining this library and all your help thinking through the issues 😄.

[peterbe]

@peterbe great idea - would something like the following work?

FANCY_COMPRESS_REMEMBERED_URLS = False  # If True, compress REMEMBERED_URLS using zlib
FANCY_MEMCACHED_USE_CAS = False  # If True, use CAS with Memcached.

Yes please!