mitmdump error: [Errno 24] Too many open files

When running mitmdump (v0.10.1) in a simple record mode, after visiting a few websites via the proxy, mitmdump eventually gets errno 24: too many open files.

e.g.

$ mitmdump -w /tmp/testtraffic --anticache -p 8080
...
53.203.15.63 GET http://s3-media4.ak.yelpcdn.com/assets/2/www/img/e41bc8917803/structural/happy_footer.png
 << 502: Error connecting to "s3-media4.ak.yelpcdn.com": [Errno 24] Too many open files
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 808, in __bootstrap_inner
    self.run()
  File "/Library/Python/2.7/site-packages/libmproxy/controller.py", line 77, in run
    self.server.serve_forever()
  File "/Library/Python/2.7/site-packages/netlib/tcp.py", line 394, in serve_forever
    request, client_address = self.socket.accept()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 202, in accept
    sock, addr = self._sock.accept()
error: [Errno 24] Too many open files

lsof -n shows about 255 open files, many of which are in the CLOSE_WAIT state. Also lots of ESTABLISHED connections. Here’s an excerpt:

Python  1817 terrylong   10u   IPv4 0x841c0e2f4ab85a1d       0t0     TCP 53.203.112.60:51956->205.180.175.183:http (ESTABLISHED)
Python  1817 terrylong   11u   IPv4 0x841c0e2f3107aa1d       0t0     TCP 53.203.112.60:51957->74.125.239.45:http (CLOSE_WAIT)
Python  1817 terrylong   13u   IPv4 0x841c0e2f489fb205       0t0     TCP 53.203.112.60:51975->74.125.239.46:http (CLOSE_WAIT)
Python  1817 terrylong   14u   IPv4 0x841c0e2f310eaa1d       0t0     TCP 53.203.112.60:51870->53.203.48.119:http (CLOSE_WAIT)
Python  1817 terrylong   15u   IPv4 0x841c0e2f310dca1d       0t0     TCP 53.203.112.60:51958->23.72.34.9:http (ESTABLISHED)
Python  1817 terrylong   16u   IPv4 0x841c0e2f489caa1d       0t0     TCP 53.203.112.60:51877->184.172.55.136:http (CLOSE_WAIT)
Python  1817 terrylong   17u   IPv4 0x841c0e2f489faa1d       0t0     TCP 53.203.112.60:51976->216.178.47.53:http (CLOSED)
Python  1817 terrylong   18u   IPv4 0x841c0e2f4ac7d205       0t0     TCP 53.203.112.60:52159->74.121.143.254:http (ESTABLISHED)
Python  1817 terrylong   19u   IPv4 0x841c0e2f489cd205       0t0     TCP 53.203.112.60:51983->74.125.239.45:http (CLOSE_WAIT)
Python  1817 terrylong   20u   IPv4 0x841c0e2f4ac18a1d       0t0     TCP 53.203.112.60:52089->205.180.175.159:http (ESTABLISHED)
Python  1817 terrylong   22u   IPv4 0x841c0e2f4ac1a205       0t0     TCP 53.203.112.60:52090->205.180.175.182:http (ESTABLISHED)
Python  1817 terrylong   26u   IPv4 0x841c0e2f4abb5a1d       0t0     TCP 53.203.112.60:52014->54.186.42.96:http (CLOSE_WAIT)
Python  1817 terrylong   28u   IPv4 0x841c0e2f31280a1d       0t0     TCP 53.203.112.60:51971->216.178.47.53:http (CLOSED)

[mhils]

Hi there,

glad to hear that it worked out for you.

I’m not sure which commit fixed that - the code in question has been mostly rewritten. We’re fairly close to the next major release ( https://github.com/mitmproxy/mitmproxy/milestones/0.11 ), so I’d suggest staying on master for now. If you should encounter any other issues, we’re happy to help here.

Cheers, Max

[monoxygen]

Ah, #281 solved it for me. I guess Mac OS X installs a system version of OpenSSL that always overrides pip’s installation.

With the master versions of netlib and mitmproxy I couldn’t reproduce the issue. I got mitmproxy up to 63 connections but they were all closed once I quit the browser on my client, as expected. Do you have any idea what commit fixed the issue, and when I can expect to see that in one of the stable releases?