FlowControl / max_messages is not working as expected

Hi,

I am trying to implement a simple task queue using Pub/Sub and I think I am running into a bug with FlowControl.

I have a few workers that need to execute a callback one message by one (without concurrency) and I use the FlowControl and max_messages=1 for that. I am expecting that every worker receive one message, process it, ack it, then pull another one.

import time
from google.cloud import pubsub
from google.cloud.pubsub import types

subscriber = pubsub.SubscriberClient()

subscription_name = 'projects/{project_id}/subscriptions/{sub}'.format(
    project_id='my-project-id',
    sub='my-subscription'
)

flow_control = types.FlowControl(max_messages=1)
    
subscription = subscriber.subscribe(subscription_name, flow_control=flow_control)

def callback(message):
    print('Received')    
    time.sleep(5)
    print('Done')
    message.ack()

future = subscription.open(callback)

future.result()

But when I publish a few messages using the following code, they are all received and processed concurrently, by a single worker.

from google.cloud import pubsub
from google.cloud.pubsub import types

publisher = pubsub.PublisherClient()

topic = publisher.topic_path('my-project-id', 'my-topic')

for i in range(1,5):	
	publisher.publish(topic, b'my-message')

Am I missing something or is this a bug ?

Thanks !

My environment details

• OS: Ubuntu 17.10
• Python: 3.6
• google-cloud-pubsub : 0.30.1

[dan-ladd]

For anyone who stumbles across this, this has been fixed here: https://github.com/googleapis/google-cloud-python/issues/7677

[Etendard7]

Thanks for your answer but the issue with the shared lock is that it doesn’t prevent a worker from receiving several messages and keeping it until the end of the processing of the first one.

Is there any way to distribute 1000 messages to 1000 workers with PubSub so that each worker receive one message ?

My use case is that I have an embarrassingly parallel task, splittable into 1000 sub-tasks and each one consume the entire memory of my worker machine.

I have tried to nack() every undesirable message but it seems that other workers can’t see them before the end of the process of the current message. That means that I cannot parallelize my task.