Performance of add_callback_threadsafe() relative to add_timeout()?

I am seeing some unexpected performance issues with BlockingConnection.add_callback_threadsafe() relative to BlockingConnection.add_timeout() in Pika 0.12.0b2, and I am wondering if this is a known issue or not.

Details follow…

I have some (working) Pika code that does the following using the BlockingConnection API:

connection.add_timeout(0.5, process_outbox)
channel.start_consuming()

Then, process_outbox() looks something like this:

def process_outbox():
    # If any outgoing messages are waiting in the "outbox", send them via channel.basic_publish()
    ....
    connection.add_timeout(0.5, process_outbox)

As a result, Pika calls process_outbox() every half second, and that’s when pending messages are sent. So far, so good.

But that approach involves up to a half-second of latency, and it spends a lot of time checking the outbox when there are no messages to send. So rewrote my code to use BlockingConnection.add_callback_threadsafe() from Pika 0.12.0b2, expecting it would be more efficient. In this new version, I just enter channel.start_consuming() right away (i.e., without setting a timeout). Then, when I need to send a message, I do something like the following:

def send(msg):
    # Put the msg in the outbox (a synchronized queue.Queue)
    ....
    connection.add_callback_threadsafe(process_outbox)

where, now, process_outbox() is as before, but it doesn’t call add_timeout():

def process_outbox():
    # If any outgoing messages are waiting in the "outbox", send them via channel.basic_publish()
    ....

I have a couple test programs that I used to compare the two approaches (both using Pika 0.12.0b2). One program queues up 3 messages to be sent ASAP, and another program recieves them.

• In the add_timeout() case, I typically see a delay of about half a second until the process_outbox() callback is called, and then the messages are all sent and received at once – as one might expect.
• In the add_callback_threadsafe() case, though, I see a delay of multiple SECONDS before the process_outbox() callback is called and the messages go out.

From what I can see in the code, add_callback_theadsafe() calls add_timeout() with a time delay of 0 seconds (via _on_threadsafe_callback()), so this is a little surprising… although the add_callback_threadsafe() call comes from another thread, so I suppose that there may be thread locking issues. (I can’t find where that is or confirm it via a profiler, though.)

My question is: Is this the expected behavior of add_callback_theadsafe()? And, are their any suggestions to make its usage more efficient?

I’ve attached a sample program that, on my system, consistently shows a delay of about 4 seconds before the callback specified by add_callback_threadsafe() is called. threadsafe_callback.py.txt

[vitaly-krugl]

That’s simple, just hit the y key when viewing a file

That’s cool, @lukebakken! I wish I found out a long time ago.

[vitaly-krugl]

@lukebakken: https://github.com/pika/pika/issues/1044#issuecomment-388488527 should never happen, because _r_interrupt file descriptor is now always registered for READ. And another reason it doesn’t happen with BlockingConnection is that the connection’s socket file descriptor is also always registered, at least for READ. In other words, there should never be a time with 0 file descriptors in the poller. With a connected BlockingConnection there should always be two active file descriptors.