Load average, LevelDB and consultancy

Hello @fergiemcdowall,

I’m Frank Rousseau, co-founder & CTO of Cozy Cloud a company that builds Cozy, a FOSS platform that makes the personal server easy to manage. It allows user to easily deploy applications on their own server. These apps collaborate together by sharing the same data store.

We included search-index in Cozy to manage the data indexation. It helps us a lot because we provide a search feature in several applications of the platform. It works fine when we index few documents like notes and file names. But when we want to make intensive indexation (like indexing a full mailbox), it crashes. Operations done via LevelDB consumes a lot of CPU and memory until the process crashes.

We would like to know if we could hire you for a few days in order to help us fixing this problem. Do you have any availabilities?

[fergiemcdowall]

@blahah Hurrah! 🎈 🍰 🎉

search-index will typically have 10-100 times as many key-value pairs as documents, depending on how long they are, and how your set your options. Also- the same key-value pair can be overwritten with every batch. If you have inserted 1.3 million into search-index, you may well have 20-30 million pairs into levelup, and possibly 100 million inserts in total (since the same keys are used in many batches).

This is a problem that it would be great to solve, and good, large datasets are surprisingly difficult to come by, so if your are OK with sharing your dataset, I would be happy to help you debug this

[blahah]

🎉 I’m going to give myself this: 🏆

I’m pretty sure the issue is in search-index, because I’ve put 3.5 million documents into a levelDB before, using levelup with leveldown backend, and memory usage was within the node default limit. I’ll double-check by putting the same 1.3 million as caused the leak above into vanilla levelup - will report back on the memory usage.