level-party leader process holding memory

env

• OS: ubuntu 20.04
• node: v16.13.1
• level-party: v5.1.1
• leveldown options: none

problem

Our level-party leader process holds up to 3GB of RAM, and refuses to release it once the activity peak passes, before being eventually killed by Linux OOM Killer (See https://github.com/inventaire/inventaire/issues/542)

It’s actually probably not a level-party-specific issue, as trying to run with only level produces the same behavior, but I didn’t know where to post: would that be a leveldown issue? Is that an expected amount of RAM for leveldb with the default options? Any clue on what we can try to address this?

Thanks for any help, I’m running short of ideas ^^’

[vweevers]

Glad to hear you found the issue, congrats!

Is this an expected behavior to be able to pass an array as a batch operation key?

Sort of, yes. It depends on the encoding. In this case the key goes through the utf8 encoding which just does String(key).

In general, encodings don’t validate input, in order to be fast and forgiving. They massage user input into a desired type. For example, db.get(Buffer.from('3')) and db.get(3) and db.get('3') all do the same thing. I’m not sure to what extent that’s by design (historically) but it’s fitting for JavaScript.

You could opt-in to stricter type checking by using TypeScript.

Is it possible to customize _serializeKey?

Encodings are applied before _serializeKey() so at that point your array key has already been converted to a string. Side note: the _serializeKey method no longer exists in abstract-level.

[maxlath]

I’m delighted to announce that this issue seems solved 😅

I went into details of the issue in the commit message here https://github.com/inventaire/inventaire/commit/4141906b7ea47fcbb6bf374104648706949b4db8 but here is a summary of what might be of interest for level-party:

An error in our code was making us build batch operations arrays with a big array mistakenly passed as an operation key. Somewhere between levelup batch function and abstratct-leveldown batch function, that array operation key was turned into a buffer, and the batch operations went on without throwing an error.

Is this an expected behavior to be able to pass an array as a batch operation key? If yes:

• That issue can be closed, and sorry for bothering
• Is there a way to optionally choose to forbid that? Is it possible to customize _serializeKey from a level-party db object, to make it reject an array passed as a key?

---

Minimum code to reproduce it:

const db = require('level-party')('/tmp/foo')

db.batch([
  {
    type: 'del',
    // This key will be converted to "foo,bar", no error will be raised
    // 'level' has the same behavior
    key: [ 'foo', 'bar' ]
  }
])
.then(res => console.log('success', res))
.catch(err => console.error('error', err))