mitmdump memory usage is always constantly growing

(orignally mentioned in #4451 , but i don’t think it’s related to the issue discussed there and should get a separate ticket)

Problem Description

i use mitmdump for interactive browsing a lot. that is, i run an mitmdump instance continuously, and use it from multiple browsers on multiple computers.

i find that mitmdump’s memory usage constantly grows, it appears as if it allocates memory for any request/response data it processes and then never frees it. note that i would expect this behaviour from mitmproxy running interactively, as it displays all the data in the UI, but mitmdump should have no reason to keep flows in memory after writing them out.

here’s a typical mitmdump memory usage graph out of my monitoring stack:

mitmdump memory usage appears as a sawtooth pattern, rising until it’s terminated. the growth rate is currently typically around 500mb/hour, it was less in the past, when it got through a whole day with under 2gb, and is higher than the rate of data actually dumped (under 1gb/day). in this graph mitmdump is manually terminated a few times, because of debugging #4451, and by a daily restart at midnight. (i also run a script that terminates it if it exceeds 3gb, to avoid it taking down my system.)

some very rare phases of non-growth (as seen in the graph above) are very unlikely related to zero traffic - thanks to websites constantly loading stuff in the background nowadays, and many browsers and tabs, it’s unlikely that traffic is ever zero.

i see some extremely rare occurences (at most once a week) of larger amounts of memory being allocated and freed again, in the expected pattern of allocating it to handle a request, and freeing it again after the request is finished.

Steps to reproduce the behavior:

1. run mitmdump for a longer period and use it, possibly with multiple browsers with a large number of tabs open
2. observe memory usage

System Information

i think it has been the case ever since i started using mitmproxy, the above graph is of current git master. (running with --set proxy_debug -vvv, but it doesn’t make a difference to the behaviour.)

[mhils]

If I add -s /tmp/script.py, initial: 55568, after 10k: 56544.

This pretty much seems like it fixes it, or would you disagree?

And the response times deteoriate pretty much (the last request is more than the double of the first one) with the number of completed requests.

This is fixed on the memoryleak branch as well, correct?

[mhils]

mitmproxy 8.1 is out now. 😃