poor performance on pipeline with buffers

Hi everyone. I’m trying to insert into redis about 15k buffers but getting poor performance. But if I try to insert same amount of data using JSON.stringify it works fast.

const pipe = redis.pipeline();
users.map(user => {
   pipe.set(`{users}:${user.id}`, UserModel.encode(user));
});
pipe.exec();
console.log("done");

on pipe.exec(); code gets freeze about for 10 seconds.

Update: UserModel is protobuf. But this does not matters: Buffer.from(JSON.stringify(user)) is also same.

[luin]

Hi @GGeorgi, Could you provide a reproducible example so I can test on my machine? I used the following code and the performance is good (v4.14.1):

const Redis = require('ioredis')
const faker = require('faker')

const redis = new Redis()

// Create a buffer from an object
function encode(obj) {
  return Buffer.from(JSON.stringify(obj))
}

// Generate some fake users
const users = new Array(150000).fill(0).map(() => (
  {
    id: Math.random() * 100000 | 0,
    name: faker.name.findName(),
    email: faker.internet.email(),
    avatar: faker.internet.avatar(),
    ip: faker.internet.ip(),
    phone: faker.phone.phoneNumber(),
    company: faker.company.companyName(),
    createdAt: faker.date.past()
  }
))

const pipe = redis.pipeline();
users.map(user => {
  pipe.set(`{users}:${user.id}`, encode(user));
});

console.log('Start #exec()')
console.time('exec')
pipe.exec();
console.timeEnd('exec')

Prints:

Start #exec()
exec: 38.897ms

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.