Google Cloud Pub/Sub triggers high latency on low messages throughput

i’m running project which publishes messages to a PubSub topic and triggers background cloud function.

I read that with high volumes of messages, it performs well, but for lesser amounts like hundreds or even tens of messages per second, Pub/Sub may yield high latencies.

Code example to publish message:

const {PubSub} = require('@google-cloud/pubsub');

 const pubSubClient = new PubSub();

 async function publishMessage() {
    const topicName = 'my-topic';
    const dataBuffer = Buffer.from(data);

    const messageId = await pubSubClient.topic(topicName).publish(dataBuffer);
    console.log(`Message ${messageId} published.`);
 }

 publishMessage().catch(console.error);

Code example of function triggered by PubSub:

exports.subscribe = async (message) => {
  const name = message.data
    ? Buffer.from(message.data, 'base64').toString()
    : 'World';

  console.log(`Hello, ${name}!`);
}

Environment details

• OS: Windows 10
• Node.js version: 8
• @google-cloud/pubsub version: 1.6

The problem is when using PubSub with low throughput of messages (for example, 1 request per second) it struggles sometimes and shows incredibly high latency (up to 7-9s or more).

Is there a way or workaround to make PubSub perform well each time (50ms or less delay) even with small amount of incoming messages?

Thanks!

[ChrisWestcottUK]

One thing that appeared to have also occurred at this time was messages published multiple times (we’re using an orderingKey):

• Without errors, we were getting back a message id when publishing. So the message publish appeared fine.
• Comparing those message id’s with the subscriber message id’s shows a mismatch (because they are message id’s of the hidden failures).
• This means we have subscribers receiving multiple messages for the same source messages but different message id’s.
• According to documentation around idempotency with Cloud Functions it’s suggested to use eventId as an idempotency key: https://cloud.google.com/blog/products/serverless/cloud-functions-pro-tips-retries-and-idempotency-in-action
• The eventId is the message id: https://cloud.google.com/functions/docs/calling/pubsub#event_structure

Doesn’t that show that it’s important to choose good idempotency keys and not follow the suggestions in the documentation? Or is there something I’m missing?

[kamalaboulhosn]

@ChrisWestcottUK The release notes for the server only track significant API/region availability/feature availability changes, not every server change that is made. There is no public tracking for all of these such changes. For tracking issues if they affect your projects, it is best to put in a support request so support engineers can follow up. This particular incident should now be resolved.