Async index updates (PUT, DELETE)

Hi Fergie, I just discovered this library and think it’s super interesting. Thank you for the work you’ve done in making this a public resource for people to learn from.

I have a question about how the JavaScript event loop is impacted during high-volume updates to the index. Is there any built-in way to run updates to the index asynchronously so that the JavaScript event loop (and the UI of my app/page) isn’t locking up during index updates?

As an analogue, the flexsearch library (which I’m also not familiar with) claims to have an async option you can use when creating an index that will apparently spin up Web Workers to perform updates, so the UI/event loop isn’t locked up during updates.

My app sometimes inserts 10,000 documents into its index at once. It’s an Electron app, so we currently manage the index in a 2nd process to prevent the UI from blocking during large index updates. But we’re considering switching to a Web Workers approach so that we can run a single process and handle index updates within a Web Worker.

Fun with event loop languages!

[aguynamedben]

Cool, thanks for the updates. To be clear, I was asking out of curiosity and to start a discussion to learn. Please don’t think I’m sending you on an errand to implement a feature request. 😃

For context, I’m building an Electron app that can index a large number of documents (100k-1m). We’re using our own logic build on top of LokiJS btrees. We have parts of our indexing/updating happening in the same process that displays the UI, so we have to be careful when doing batch operations. That’s why I was asking about this.

We’ll probably move more of the indexing to a separate process, so that the UI thread isn’t doing any CPU-bound work. This is similar to the concept of using Web Workers. The UI would send [query or batch update] to the [Web Worker or background process] and not be locked up during the response.

Thanks for the feedback, there are tons of interesting links to look through. 🙏 Feel free to close the issue unless you all want to keep the discussion open for a specific reason.

[fergiemcdowall]

@eklem kudos for getting it to work- thats really interesting!