Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

KafkaProducer fails when many sockets are opened

See original GitHub issue

When using KafkaProducer in an environment with over 1024 open sockets we get the following issue:

File "/srv/.../lib/python2.7/site-packages/kafka/producer/kafka.py", line 248, in __init__
self.config['api_version'] = client.check_version()
File "/srv/.../lib/python2.7/site-packages/kafka/client_async.py", line 639, in check_version
self.poll(future=f)
File "/srv/.../lib/python2.7/site-packages/kafka/client_async.py", line 367, in poll
responses.extend(self._poll(timeout, sleep=sleep))
File "/srv/.../lib/python2.7/site-packages/kafka/client_async.py", line 402, in _poll
ready, _, _ = select.select(fds, [], [], timeout)
ValueError: filedescriptor out of range in select()

AFAIK this is because the select() call is limited to FD_SETSIZE being 1024 on Linux.

Issue Analytics

State:
Created 7 years ago
Comments:10 (4 by maintainers)

Top GitHub Comments

1reaction

dpkpcommented, Mar 29, 2016

thanks for bug report! we’ll need to get more sophisticated here. one semi-related improvement is to add support for closing idle sockets (Issue #548). the real fix here may be to check for select alternatives (epoll / poll / etc).

0reactions

dpkpcommented, Apr 9, 2016

I put up PR #640 to migrate to the selectors module and remove the explicit use of select. Please let me know if you have comments or have a moment to test whether this fixes your issue.