question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Dev server hangs for 1-2min+ when lots of small files in project directory

See original GitHub issue

Is your feature request related to a problem? Please describe. I added ~80mb worth of tiny files (small icon .pngs and .jsons) to my static assets paths. When dev server started, the first request took ~2 minutes to complete. After that it was usually ok for a while, but sometimes this 2min delay reoccurred, making developing on the npm run dev very difficult

I am not 100% sure what the cause of this problem was. I suspect it is the file watcher of the dev server, but it could be something else. I read some source code and it seems like cheap-watch is being used to watch files on the disk. I never found a reference to this watcher in the documentation, and there is no way to configure it from svelte.config.cjs. There is the Vite config, which in turn passes some options to a chokidar package (which is the watcher that Vite uses), and some people in the Discord seemed to think that I should change my Vite config to add the files to the ignore list. When I played around with these configs, it seemed to have no effect at all, and me finding out about cheap-watch seems to support my theory that cheap-watch is indeed the one watching my files.

I solved the problem by moving my static assets outside my project root, to a sibling directory.

Describe the solution you’d like Watch files based on a watched_dirs glob whitelist (do not watch anything unless explicitly told so) in svelte.config.cjs along with sane defaults, or watch everything in the local directory, but add user configurable ignored list of globs to ignore.

Also, as a sidenote, it would be nice to have some kind of --verbose option for the dev server, so when you’re running it you’d know what’s taking so long for it to respond.

Describe alternatives you’ve considered One solution is to completely ignore the configured assets path on dev server, although this would of course have the side effect of not triggering the watcher when static assets are modified. But since no one solution fits everyone, I’d like this to be configurable with a configuration option.

Or, just optimize the file watcher so well that it can easily handle tens of thousands of small files without the server hanging.

How important is this feature to you? For now, I can make do with moving my static assets to a sibling directory of the project directory, so I am able to move on, but I spent plenty of hours debugging this, so improvements would be very good in order to attract people who don’t want to dive into source code to debug.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:17 (9 by maintainers)

github_iconTop GitHub Comments

1reaction
deepmousecommented, May 10, 2021

Should I open a separate issue about the --verbose flag? It would really help me debug this problem.

0reactions
deepmousecommented, May 10, 2021

This is getting even more weird. I now moved the static assets back to the sibling repo, and this time it didn’t help… My dev server is now doing this even when the assets outside the project directory.

Read more comments on GitHub >

github_iconTop Results From Across the Web

webpack-dev-server breaks on changing folder structure or ...
In my opinion this is a bug because it was working in older versions of webpack. What is the current behavior? When I...
Read more >
Opening many small files on NTFS is way too slow
I've been testing that part on 500k files, and the first step was just to iterate a directory which has around 45k directories...
Read more >
rsync is extremely slow to synchronize a large amount of ...
The rsync is used to synchronize the files from a /home/user/folder_with_subfolders to an NFS mounted folder /home/user/mountpoint .
Read more >
How many files in a directory is too many? (Downloading ...
Performance varies according the the filesystem you're using. FAT: forget it :) (ok, I think the limit is 512 files per directory); NTFS:...
Read more >
Change freeze: pros, risks, and project management tips
In short, a change freeze supports project completion and protects business operations. Ultimately, it benefits two main parties: your business ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found