High Latency To Publish

Hello,

I am using the publish function in a basic way:

const gPubsub = require('@google-cloud/pubsub');
const pubsub = gPubsub({
      projectId: 'projectId',
    });
const publish = (message, cbkPublish) => {
      if (!message.topic)
        return cbkPublish('No PubSub topic specified');
      const topic = message.topic;
      delete message.topic;
      const publisher = pubsub.topic(topic).publisher();
      const buff = Buffer.from(JSON.stringify(message));
      publisher.publish(buff)
        .then(() => cbkPublish(null))
        .catch(err => cbkPublish(err));
    };
    return callback();
  }

The problem is that I am experiencing a long time to publish, for a buffer length of 352 it takes 3608 ms to publish. Is there some settings to publish faster ? Am I doing something wrong ?

Environment Details:

Node: 8.1.4
google-cloud/pubsub: 0.16.5

[troystackforce]

I experienced a similar issue recently. Our use case is to publish a single message to a pubsub topic from within a web client request/response cycle (an express route).

I wrote this test script to evaluate the time it takes to publish a message.

const PubSub = require('@google-cloud/pubsub');

const topicName = 'test-publish-speed-topic';
let savedTopic;
let savedPublisher;
const maxMessages = 1;
const maxMilliseconds = 10;
let totalDuration;
let loopCtr;

const getTopic = async () => {
  if (!savedTopic) {
    const pubsub = new PubSub();
    savedTopic = (await pubsub.topic(topicName).get({autoCreate: true}))[0];
  }
  return savedTopic;
};

const getPublisher = async () => {
  if (!savedPublisher) {
    const topic = await getTopic();
    savedPublisher = topic.publisher({
      maxMessages,
      maxMilliseconds,
    });
  }
  return savedPublisher;
};

const deleteTopic = async () => {
  const topic = await getTopic();
  if (topic) {
    await topic.delete();
    console.log(`topic ${topicName} deleted`);
    return;
  }
  console.log(`topic ${topicName} not found`);
};

const publish = async (data) => {
  const publisher = await getPublisher();

  const start = Date.now();
  await publisher.publish(data);
  const end = Date.now();
  const duration = end - start;
  totalDuration += duration;

  console.log(`publish: dur: ${duration}ms avg: ${totalDuration / loopCtr}ms`);
};

const run = async () => {

  // This will auto-create the topic. Do this before the run
  // so topic creation doesn't influence the results.
  await getTopic();

  // Exercise the publish function and console.log timing results
  totalDuration = 0;
  loopCtr = 1;
  for (; loopCtr <= 1000; ++loopCtr) {
    const data = {"foo": "bar"};
    const dataAsBuffer = Buffer.from(JSON.stringify(data));
    await publish(dataAsBuffer);
  }

  // Cleanup
  await deleteTopic();

};

run();

It takes ~120ms to publish a message using the above test script in cloud shell which is longer than we were hoping for. I reached out to the gcp slack channel and tscanausa tested it on his end using this curl command:

PROJECT='your-project'; TOPIC='your-topic'; curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" -X POST https://pubsub.googleapis.com/v1/projects/$PROJECT/topics/$TOPIC:publish -d @data.json -H "Content-Type: application/json" -s -w @curl_format.txt

curl_format.txt

\n
  time_namelookup:      %{time_namelookup}\n
  time_connect:         %{time_connect}\n
  time_appconnect:      %{time_appconnect}\n
  time_pretransfer:     %{time_pretransfer}\n
  time_redirect:        %{time_redirect}n
  time_starttransfer:   %{time_starttransfer}\n
                     ------------\n
          time_total:   %{time_total}\n
\n

data.json

{  
   "messages": [{
      "data": "SGVsbG8sIFdvcmxkIQ=="
  }]
}

with similar results. I’m happy to help profile the node client if you think it will help? I tried batching 100 messages at a time and the publish time didn’t vary much so it appears to take around 120ms whether I publish 1 message or x messages up to some value of x.

[kir-titievsky]

Yes, please, @aayusharora – don’t remove the settings. Set them explicitly. As I said, the default is to batch.