Code snippets crash Heroku instances[🐛]

🐛 Bug report

Description

When requesting several code snippets in a short time or in parallel, the following error can occur and eventually crash a Heroku instance:

2021-04-11T18:15:43.315208+00:00 app[web.1]: node:events:346
2021-04-11T18:15:43.315236+00:00 app[web.1]:       throw er; // Unhandled 'error' event
2021-04-11T18:15:43.315237+00:00 app[web.1]:       ^
2021-04-11T18:15:43.315237+00:00 app[web.1]: 
2021-04-11T18:15:43.315238+00:00 app[web.1]: Error: EMFILE: too many open files, open 'frontend/src/app/user-details/user-details.component.spec.ts'
2021-04-11T18:15:43.315239+00:00 app[web.1]: Emitted 'error' event on ReadStream instance at:
2021-04-11T18:15:43.315239+00:00 app[web.1]:     at emitErrorNT (node:internal/streams/destroy:188:8)
2021-04-11T18:15:43.315240+00:00 app[web.1]:     at emitErrorCloseNT (node:internal/streams/destroy:153:3)
2021-04-11T18:15:43.315241+00:00 app[web.1]:     at processTicksAndRejections (node:internal/process/task_queues:81:21) {
2021-04-11T18:15:43.315241+00:00 app[web.1]:   errno: -24,
2021-04-11T18:15:43.315242+00:00 app[web.1]:   code: 'EMFILE',
2021-04-11T18:15:43.315242+00:00 app[web.1]:   syscall: 'open',
2021-04-11T18:15:43.315243+00:00 app[web.1]:   path: 'frontend/src/app/user-details/user-details.component.spec.ts'
2021-04-11T18:15:43.315243+00:00 app[web.1]: }
2021-04-11T18:15:43.404862+00:00 heroku[web.1]: Process exited with status 1
2021-04-11T18:15:43.468277+00:00 heroku[web.1]: State changed from up to crashed

These crashes occured when running the e2e test suite of the upcoming v8.2.0 of juice-shop-ctf as it uses the demo instance on Heroku as its counterpart to retrieve challenges and now also the code snippets.

Implementing a simple cache in the Juice Shop route to avoid multiple reads of the same challenge snippet did not solve the issue. The CTF tool’s tests still overwhelm Heroku instances with open files, probably due to the inefficient current implementation in vulnCodeSnippet.ts.

Release of CTF extension v8.2.0 will be delayed until this problem is fixed on Juice Shop web app side, to avoid artificially causing even more crashes of the demo instance.

Is this a regression?

Yes, the previous version in which this bug was not present was: v12.6.1 where Code Snippets were not implemented yet.

🌳 Your Environment

https://demo.owasp-juice.shop

[the-pro]

What if we write a manual fileSniffer and use graceful-fs instead of fs. I read this about the graceful-fs here:

The goal is to trade EMFILE errors for slower fs operations. So, if you try to open a zillion files, rather than crashing, open operations will be queued up and wait for something else to close.

[bkimminich]

The change by @the-pro did the trick! Juice Shop CTF test suite no longer kills the Heroku instance! 😎👍