mitmdump: temporarily running out of diskspace corrupts dumpfile

Problem Description

ofcourse the world would be a better place if everybody always responsibly managed their filesystem space, but we have to face that fact that mitmdump, as a non-interactive dumping tool, is often going to be the program that fills up the last available block on a filesystem.

currently it deals with that situation very badly:

• it keeps running, and keeps trying to write to the file
  • which in part is good, because it will resume saving data when space becomes available again
• the last flow it tried to write is left in the file half-written and truncated, which due to the length-prefixed data format leads to the file being corrupt and not loadable with the provided tools.

due to this situation i already learned more about the file format than i ever wanted to 😉 the key to recovering data in such a corrupted file is:

grep -oE --binary-files=text --byte-offset '[0-9]+:7:version;'

(and i wrote a script that automates the process)

but it would be better to fix this in mitmdump itself. suggestions:

• on a failed write, truncate the file back to it’s previous length
• on a failed write, exit with an error
• maybe provide a tool to recover data from corrupted files

Steps to reproduce the behavior:

log more data than is good for you

System Information

any version i ever used

[mhils]

mitmproxy now at least exits on a full disk. Truncating the file back to its previous length isn’t trivial unfortunately, calling f.truncate(pos) on a full disk just raises another disk full error. Not sure what’s the best way to approach this as we can’t easily reopen from within FlowWriter. 🙃

[r00t-]

the baseline should be to not ignore the error, and exit. creating a warning in advance could be nice to have on top, as also pointed out by mhils:

Optionally we can also consider checking shutil.disk_usage on top and exit at 99% disk usage already