socket.timeout on heartbeat (or any other write)

Hi,

I used kombu with the py-amqp transport and I have the following issue:

I have consumer code that looks like:

while True:
      try:
          conn.heartbeat_check()
          try:
               conn.drain_event(timeout=mytimeout)
          except socket.timeout:
               pass
          do_special_stuff()
      except conn.recoverable_channels_errors + conn.recoverable_connection_errors:
          do_repair_connection()

So I expect that ‘do_special_stuff’ is called every 'mytimeout ’ or the connection have been closed. But sometimes the heartbeat_check() will write a packet to the socket to send the heartbeat frame and if something bad occurs at this moment on the tcp connection, the application can stuck until the system tcp timeout is reached and then heartbeat_check() raise socket.timeout.

So I would like to control the timeout of the writing of heartbeat frame (and more globally any other frame we write on the socket when we declare or publish something)

After some digging into the kombu/py-amqp code, I haven’t found a way to set the socket timeout of the connection used to write that (or any other) frame to the socket.

Does my needs make sense to allow a default socket timeout per connection or something like that ? and that just need to be implemented, or I have missed something ?

(Note: I would like to avoid to change the default python socket timeout or set a different system tcp timeout)

Cheers,

[stuartspotlight]

Was this issue successfully resolved? I think I am getting a similar issue running kombu 4.1.0 where kombu is disconnecting on a publish command with the following error:-

  W_FORCE_CONNECT.format(attr=attr)))
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/nanowire_plugin/__init__.py", line 447, in send_to_next_plugin
    'max_retries':10})
  File "/usr/local/lib/python3.6/site-packages/kombu/messaging.py", line 181, in publish
    exchange_name, declare,
  File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 494, in _ensured
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/kombu/messaging.py", line 203, in _publish
    mandatory=mandatory, immediate=immediate,
  File "/usr/local/lib/python3.6/site-packages/amqp/channel.py", line 1745, in basic_publish_confirm
    self.wait(spec.Basic.Ack)
  File "/usr/local/lib/python3.6/site-packages/amqp/abstract_channel.py", line 79, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 471, in drain_events
    while not self.blocking_read(timeout):
  File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 477, in blocking_read
    return self.on_inbound_frame(frame)
  File "/usr/local/lib/python3.6/site-packages/amqp/method_framing.py", line 55, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 480, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
TypeError: 'NoneType' object is not subscriptable

This is failing to reconnect even after attempting to reconnect 10 times:-

            producer.publish(send_payload, exchange='', routing_key=next_plugin, retry=True, 
                             retry_policy={'interval_start':1,
                                           'interval_step':2,
                                           'interval_max':10,
                                           'max_retries':10})

Also worth noting is that I get the following warning when kombu attempts to reconnect:-

/usr/local/lib/python3.6/site-packages/amqp/connection.py:292: AMQPDeprecationWarning: The .transport attribute on the connection was accessed before
the connection was established.  This is supported for now, but will
be deprecated in amqp 2.2.0.

Since amqp 2.0 you have to explicitly call Connection.connect()
before using the connection.

I assume this warning will disappear when we move to kombu 4.2 later today but is there any word on the publish disconnect error?

[auvipy]

closing as fixed