Inadvertently removed the `buffer/asBuffer` optimization

Continuing from https://github.com/Level/encoding-down/pull/17#issuecomment-337314850. I found this issue trying to make sense of the buffer option on encodings, and eventually traced it back to levelup.

In levelup@1, we had an optimization for string-based encodings like json:

exports.json = {
  encode: JSON.stringify,
  decode: JSON.parse,
  buffer: false,
  type: 'json'
}

The important property here is buffer: false. This told levelup to prefer to fetch a string from the underlying store, rather than a Buffer. The underlying store would receive { asBuffer: false } in get options and similar places.

This is how the code looked:

LevelUP.prototype.get = function (key_, options, callback) {
  options.asBuffer = this._codec.valueAsBuffer(options)

  this.db.get(key, options, function (err, value) {}
}

When we removed the encoding logic from levelup (#458), the buffer/asBuffer logic was also removed and did not get a replacement elsewhere.

So since levelup@2, underlying stores always see { asBuffer: true }. The means we incur a conversion cost, sometimes even double conversion:

• leveldown always returns keys and values as Buffers (with an additional memcpy cost, but let’s leave that aside for a moment)
• memdown as well will convert strings to Buffers (if they were stored as strings, which is true for the json encoding at least)
• The encodings convert Buffers back to strings (in the case of json it implicitly does JSON.parse(x.toString())).

[juliangruber]

Since this is about encodings, and needs to know about encodings, can we solve this in encoding-down? This part of the stack knows which option is best, and should then be able to pass it to the underlying store.

Great find, @vweevers!

[vweevers]

Before we decide and implement a fix, it would be nice to have some failing tests for this behavior.

👍 I’ll rewrite what I have into a test for encoding-down.