[httpcache] FilesystemCacheStorage file organization

• Situation
• I use FilesystemCacheStorage, and have saved about 1M requests, 30GB data (not sure, I’m still waiting for counting …). And it can’t proceed reporting no disk space, but I still have 10+GB on disk.
• Interesting thing is that I can’t even use auto completion now.

-bash: cannot create temp file for here-document: No space left on device

• Comment
• I don’t know if it is a valid use case, I use filecache to save call to APIs to allow reprocessing the response as needed (to lower burden on service).
• I think it’s possible that fs has run out of inode. And the current implementation seem to create file for each request. Maybe can combine cache in save file.
• Or maybe the solution is to use database?

[Gallaecio]

@larryxiao Is it OK to close this issue?

If you would like to keep it open as a feature request, it might be a good idea to be more explicit about the desired change that, once implemented, should allow to close this issue.

[setop]

Cache eviction might not be the responsibility of a running crawler.

Anyway, out of a crawer, a user can run a job to purge cache based on various policy (oldest first, response cache policy, last access).

eg, for “oldest first” policy, one can run :

find httpcache -maxdepth 3 -mindepth 3 -type d -mtime +30d with -delete flag.

A hardest one would be “least recent access first” policy as access time is not an information stored by scrapy caching module and rarely stored by filesystems (atime) for performance reason.