Default URL redirects are cached too aggressively

Details

• Read the Docs project URL: https://readthedocs.org/projects/pelorus/

Browser: chrome Version 103.0.5060.134 (Official Build) (arm64) OS: macOS 12.4

Steps

1. Set up automation to make latest release the default url:
   • Match: SemVer versions
   • Version type: tag
   • Action: set version as default
2. Set up a github incoming webhook
3. Tag a release
4. View the docs at /
5. Tag another release within 24 hours
6. Verify that the default version in Advanced Settings / Global Settings / Default version points to the release from the previous step
7. View the docs at /

Expected Result

I am redirected to /en/$tag_from_step_5

Actual Result

Chrome loads the 302 FOUND from cache, redirecting me to /en/$tag_from_step_3.

Analysis

Here’s the response headers, I assume from the initial request at step 4:

Response Headers (some omitted, if I don't know how private they are)

cache-control: public, max-age=86400
cf-cache-status: MISS
content-language: en
content-length: 0
content-type: text/html; charset=utf-8
date: Tue, 09 Aug 2022 18:33:36 GMT
expires: Wed, 10 Aug 2022 18:33:36 GMT
location: https://pelorus.readthedocs.io/en/v1.6.0/
referrer-policy: no-referrer-when-downgrade
server: cloudflare
vary: Accept-Language, Cookie, Accept-Encoding
x-content-type-options: nosniff
x-rtd-domain: pelorus.readthedocs.io
x-rtd-project: pelorus
x-rtd-project-method: subdomain
x-rtd-redirect: system
x-rtd-version-method: path
x-served: Django-Proxito

Chrome caches redirects, but will honor cache-control headers on the redirect response. If I understand how this works, max-age=86400 means the redirect will be cached for 24 hours.

Perhaps conditional cache headers should be used for the top-level URL, instead of max-age in cache-control. Last-Modified or ETag (generated based on the default version) should work, if cloudflare can handle those. Note: The cache for the Location (the actual version, /en/$version) can stay the same! This isn’t about that.

I do not believe this is related to #9167. The tagged version is fresh and served correctly. According to chrome, it doesn’t even make the request because of the cache-control header. That issue says “a few minutes”-- the redirect is still pointing to the older version, despite the automation changing the settings 1 hour and 16 minutes ago at time of writing.

I’m not sure how this affects other browsers. Perhaps they’re less aggressive at caching redirects.

If needed, I can try to make a smaller (perhaps even automated) reproduction of the bug.

[ericholscher]

The Browser Cache TTL sets the expiration for resources cached in a visitor’s browser. By default, Cloudflare honors the cache expiration set in your Expires and Cache-Control headers but overrides those headers if:

• The value of the Cache-Control header from the origin web server is less than the Browser Cache TTL Cloudflare setting.

From: https://developers.cloudflare.com/cache/about/edge-browser-cache-ttl/

So I think we can lower the default, and set it in our application.

[humitos]

@stsewd

I think we should reduce the max-age value, maybe 10 or 5 minutes is enough, cc @ericholscher

Are we setting this value at the application or is it done at Cloudflare? Can we reduce the max-cache only on redirect responses but keep it as is for regular pages?