Pushing 400k messages creates 10m messages to pubsub

I tried to push avro file contents to pubsub (1 record/message). The file had 429 077 records but after the push pubsub graph showed about 10 million messages. Also logged the publish calls and it was called the correct amount of times (429 077). Code is below and it’s highland stream wrapped to promise.

export const avroInBucketToPubSub: AvroInBucketToPubSub = ({
  exportFile,
  inputTopicName,
  validator
}) => {
  const pubsub = new PubSub();
  return new Promise((resolve): void => {
    const inputTopic = pubsub.topic(inputTopicName);
    hl(exportFile.createReadStream().pipe(new streams.BlockDecoder()))
      .filter(row => validator.isValid(row))
      .map(JSON.stringify)
      .map(Buffer.from)
      .each(buff => inputTopic.publish(buff))
      .done(resolve);
  });
};

Environment details

• OS:
• Node.js version: v10
• npm version: 6.11.3
• @google-cloud/pubsub version: 0.30.1

Steps to reproduce

1. push over 100k messages without waiting reply

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[anthonymartin]

fair, that should be fixed when publisher flow control is implemented as per above discussion

[kamalaboulhosn]

So there is no such thing as publisher-side flow control for cloud that would help the user here. We are in the process of adding publisher flow control to our client libraries and I think there is still outstanding work on node for this. I think that’s the next step to take here.