question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Use url-normalize to normalize requests before creating cache key

See original GitHub issue

I’m having an issue where cache doesn’t persist properly. I’m using python3 and the sqlite backend.

if I open the cache sqlite file, the urls table is empty, and the responses table has 136 items, even though my script only makes 56 requests which don’t change across runs.

I think what’s happening here is that (since python dicts are not ordered) the order of the request parameters is slightly different each time, which causes the GET parameter order to be different, and hence the cache key to be different too.

Issue Analytics

  • State:closed
  • Created 7 years ago
  • Comments:5

github_iconTop GitHub Comments

1reaction
elad661commented, Jul 22, 2017

https://github.com/reclosedev/requests-cache/blob/master/requests_cache/backends/base.py#L225

It just takes the URL as-is and computes its hash. If the parameters changed orders (which they might, since requests uses a dict for the request parameters and python dicts are not ordered), then the hash would be different.

It’d be better to change this to split off the parameters, sort them, and then compute the hash, so it’d be consistent across runs.

0reactions
JWCookcommented, Feb 25, 2021

I think this one is still valid. Would someone like to submit a PR to normalize request params when creating cache keys?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Use url-normalize to normalize requests before creating ...
I'm having an issue where cache doesn't persist properly. I'm using python3 and the sqlite backend. if I open the cache sqlite file, ......
Read more >
Controlling the cache key - Amazon CloudFront
Normalizes the header to Accept-Encoding: gzip and includes the normalized header in the cache key. If the edge location has a Gzip compressed...
Read more >
Cache Keys - requests-cache 0.9.7 documentation
Normalize and remove ignored parameters from request URL, body, and headers. This is used for both: Increasing cache hits by generating more precise...
Read more >
Normalized Caching | urql Documentation
A normalized cache seeks to turn this denormalized JSON blob back into a relational data structure, which stores all entities by a key...
Read more >
Demystifying Cache Normalization
Apollo Client provides APIs for building rich client applications by normalizing, caching, and making sense of complex GraphQL data.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found