Pub/Sub access to message ack id in the subscribe use case

When pulling messages, we get ReceivedMessage that contains the ack id that can then be used to ack multiple messages at once using the ModifyAckDeadlineRequest.

However, on the subscribe side all we get is PubsubMessage and AckReplyConsumer, without access to the ack id of the message. So, this means that we cannot ack messages in batch.

@garrettjonesgoogle

[kamalaboulhosn]

@unoexperto Relatively speaking, I expect the AckReplyConsumer is going to be much smaller than the messages you are buffering themselves, so I don’t think holding on to them is going to be a major contributing factor to OOMs. However, holding on to messages for an hour in a subscriber client and only acking after a batch is flushed is an anti-pattern for Cloud Pub/Sub. You want to try to ack messages very quickly. The longer you wait to ack, the more risk there is of duplicate delivery, which would also waste resources. You’d be better off writing messages to some persistent storage locally when you receive them, acking them, and then flushing the data every hour. You could do this using Apache Beam/Cloud Dataflow.

[unoexperto]

@pongad @kamalaboulhosn This is very limiting factor. Could you please suggest how one can postpone acking bunch of messages? I’m using PubSub as durable storage of messages which I batch into 1 hr chunks and send elsewhere. I don’t want to accept messages until I’m sure my consumer got the batch. I could of course hold on to instances of AckReplyConsumer for each message in the batch that I haven’t processed yet but it poses risk of OOM exception as you point out in documentation.

Because the client library repeatedly extends the acknowledgement deadline for backlogged messages, those messages continue to consume memory, CPU, and bandwidth resources. As such, the subscriber client might run out of resources (such as memory). This can adversely impact the throughput and latency of processing messages.

What would you suggest?