Inserting in a loop does not insert all keys.

I was troubleshooting an issue with a wrapper I made, and finally realized this seems to be a native level issue. When inserting 100,000 records in a level database, some keys are missing. Exactly 433 to be precise.

Here is my test code

const Level = require('level');
const db = new Level('./data/points');
const test = new Map();

(async function() {
  console.time('100000 Inserts');
  for (let i = 0; i < 100000; i++) {
    await db.put(`test${i}`, 'simple string');
  }
  console.timeEnd('100000 Inserts');
  await fetchEverything();
  console.log(`${test.size} keys loaded.`);
  for(let i = 0; i < 100000; i++) {
    if(!test.has(`test${i}`))
      console.log(`Missing test${i}`);
  }
}());

const fetchEverything = async function() {
  return new Promise((resolve) => {
    const stream = db.keyStream();
    stream.on('data', (key) => {
      db.get(key, (err, value) => {
        if (err) {
          console.error(`Error loading: ${err}`);
        }
        test.set(key, value);
      });
    });
    stream.on('end', () =>
      resolve()
    );
  });
}

Every time I run this, it says it inserts 100000, but when I fetch everything… I only get 99567 keys loaded. The strangest thing is, all those keys start with test9, such as test9, test99, test999. I have tried this multiple times, and always get the exact same number of keys, never more or less.

I’m on level 3.0.0, running under Windows 10 (Fully updated to Creators and all further patches). No errors are logged when fetching the values. Issue also happens on Ubuntu Server 16.04. Node versions are 8.10 and 9.9 respectively.

[eslachance]

Ah, I see what’s happening. I did not realize that the stream end event happened before all get promises were resolved. I would assume using Promise.all(fetches).then(()=> resolve()) would work, assuming fetches is an array of all the db.get() promises. Thanks for the information, guys!

[juliangruber]

@ralphtheninja’s observations are correct, just adding another alternative:

const {promisify} = require('util')
const read = promisify(require('stream-read'))

const fetchEverything = async () => {
  const stream = db.createReadStream()
  let key, value
  for (({ key, value }) = await read(stream)) test.set(key, value)
}