Slow startup with many static files

When multiple Django WSGI processes start in parallel accessing thousands of static files via network file system, it can take quite a while to seed the WhiteNoise.files static files cache, during which time the Django app can’t yet service inbound requests. (For example, we have server instances with 6 Django processes running Whitenoise 4.1.3 accessing ~4,000 static files via EFS (AWS-flavored NFS) which takes roughly 45 seconds to finish scanning the staticfiles dirs and begin accepting requests.)

I see the WHITENOISE_AUTOREFRESH setting to skip the cache initialization, but that’s labeled not for production use, and clearly adds ongoing overhead, as it stats every file at request time.

I wonder if there’s another configuration option we’re missing, if we’re using the tools incorrectly, or if you’ve considered adding another mode, where the WhiteNoise.files cache is lazy loaded incrementally as each file is requested?

[evansd]

I wouldn’t be totally opposed to adding support for it, but it would depend on how much complexity it adds and, specifically, how well-isolated that complexity is from everything else. This feels like one of those situations where adding basic support which works but with a few rough edges would be relatively easy, whereas getting something really solid which handles all the edge cases would be a lot more work.

I’d certainly be happy to reference your use case in the documentation though and link to any code you have. I’d say, get something that works for you by subclassing the middleware and then we can decide from there whether it looks worth trying to include in the main project.

[jbiggar]

Thanks for the thoughtful response @evansd.

Do you think a build/deployment-time-regenerated and initialisation-time-loaded pickle file cache is something you’d want to support in the main project? We’ll probably head down the path you suggest, but wouldn’t bother with a full PR to the upstream project if that’s not a direction you’d want to head.