poor interactive performance / hanging requests

i use mitmproxy for interactive browsing a lot. that is, i run an mitmdump instance continuously, and use it from multiple browsers on multiple computers.

recently this started to be a rather frustrating experience. i suspect it’s related to the new proxy core, or it could be related to debian upgrading to python 3.9? sadly it turned out non-trivial to roll this back.

initially after mitmproxy startup, everything works smoothly, but then suddenly stalls, many requests hang for a very long time until mitmproxy returns a response. sometimes this starts almost immediately, sometimes after hours, it possibly recovers by itself, but usually i give up waiting and restart mitmproxy. (most times the requests seem to do finish, but only after many minutes.) this is probably most easily reproduced with resource-heavy sites, like facebook.com, openstreetmap.org, …

in netstat, i don’t see mitmproxy having many connections to the destination servers open, so i suspect it’s an internal limitation. ( i typically check with $ netstat -ntp | grep EST.*python | grep -v :8080 | awk '{print $5}' | sort | uniq -c | sort -n)

my initial suspicion was that there is a limit on the number of concurrent connections, and that that pool is used up by connections that are in some invalid state and/or waiting for a timeout.

i already found a per-connection.address] limit at https://github.com/mitmproxy/mitmproxy/blob/dc6c5f55cd25236e9469c24b85c8cafd29573281/mitmproxy/proxy/server.py#L84 and increased it significantly, but that did not lead to consistent improvements. (i also added a warning on if self.max_conns[command.connection.address].locked():, but it doesn’t seem to trigger.)

can you suggest what else i could do to diagnose this? is there another connection limit besides max_conns in proxy? is there existing code/facilities to introspect/dump the currently open incoming connections and their state? an option to log connection-level tracing info?

System Information

Paste the output of “mitmproxy --version” here.

(venv) user@host[~/mitmproxy]$ mitmproxy --version
Mitmproxy: 7.0.0.dev (+332, commit aebc40c)
Python:    3.9.1+
OpenSSL:   OpenSSL 1.1.1i  8 Dec 2020
Platform:  Linux-5.8.0-1-amd64-x86_64-with-glibc2.31

[mhils]

I was able to reproduce this locally, this was quite a regression introduced in #4458. 😛 #4459 has the fix!

[mhils]

We do indeed kill open connections after 10 minutes of total inactivity. In this case it sounds like the client is still waiting for a response for some reason. A log with --set proxy_debug -vvv would go a long way in identifying the root cause. Feel free to shoot that by email if you have sensitive stuff in there.