Piping mitmdump output to another program and quitting the latter early results in BrokenPipeErrors

Problem Description

Piping mitmdump output to another program and exiting that latter program before it has read all of mitmdump’s output produces a lot of BrokenPipeError: [Errno 32] Broken pipe errors.

Steps to reproduce the behavior:

1. Capture a non-trivial amount of traffic: mitmdump -w outfile; the nature of the traffic (HTTP/HTTPS/WebSocket) does not seem to matter. The outfile has to be large enough for the next command to produce enough output that the pipe buffer gets backlogged.
2. Print the contents of the dump file and pipe it to something like less: mitmdump -nr outfile | less
3. Quit less with the q key.
4. mitmdump produces a traceback similar to the following for every record in the outfile that has not been consumed by less prior to quitting:

Addon error: Traceback (most recent call last):
  File ".../lib/python3.8/site-packages/mitmproxy/addons/termlog.py", line 31, in log
    click.secho(
  File ".../lib/python3.8/site-packages/click/termui.py", line 548, in secho
    return echo(message, file=file, nl=nl, err=err, color=color)
  File ".../lib/python3.8/site-packages/click/utils.py", line 273, in echo
    file.flush()
BrokenPipeError: [Errno 32] Broken pipe

Addon error: Traceback (most recent call last):
  File ".../lib/python3.8/site-packages/mitmproxy/addons/dumper.py", line 263, in websocket_message
    self.echo(f.message_info(message))
  File ".../lib/python3.8/site-packages/mitmproxy/addons/dumper.py", line 70, in echo
    click.secho(text, file=self.outfp, **style)
  File ".../lib/python3.8/site-packages/click/termui.py", line 548, in secho
    return echo(message, file=file, nl=nl, err=err, color=color)
  File ".../lib/python3.8/site-packages/click/utils.py", line 273, in echo
    file.flush()
BrokenPipeError: [Errno 32] Broken pipe

Attempted workarounds and possible solution

The obvious workaround is to first make the second program consume everything and only then quit. For example, with less, pressing G to scroll to the bottom of the output before quitting. However, since less keeps everything in memory, this may not be feasible for large dump files.

Killing metadump with ^C after quitting less does not seem to help.

For this particular use case of displaying file information, the ideal solution would be to simply exit on a BrokenPipeError. This would also mean that the rest of the dump file would not be processed, needlessly wasting CPU time. But the general case is probably rather to disable all further output to stdout but keep processing (e.g. if a dump file is being filtered into another file).

System Information

$ mitmproxy --version
Mitmproxy: 5.1.1 (+2, commit a8ca63f)
Python:    3.8.2
OpenSSL:   OpenSSL 1.1.1g  21 Apr 2020
Platform:  Linux-5.6.0-1-amd64-x86_64-with-glibc2.29

This is a current Debian sid system. The (+2, commit a8ca63f) info on the mitmproxy version is a bug (#3987) and can be disregarded; this is a clean 5.1.1 installation.

[JustAnotherArchivist]

@myfingerhurt I’m not sure about the other things you mentioned, but about this:

mitmweb mitmweb -s mitm_anatomy.py 2>&1 > /tmp/mitm.log did save both stderr and stdout to mitm.log but only flushed to the file after terminated. I needed the real-time log for tail -F to work with another program.

You may want to try setting the PYTHONUNBUFFERED environment variable to a non-empty value.

[myfingerhurt]

This is funny though. Pipe is the most fundamental and most wanted and should work flawlessly as we all expected.

mitmweb --version
Mitmproxy: 8.0.0
Python:    3.8.10
OpenSSL:   OpenSSL 1.1.1n  15 Mar 2022
Platform:  Linux-5.4.0-1055-raspi-aarch64-with-glibc2.29

mitmweb -s mitm_anatomy.py | tee /tmp/mitm.log this didn’t work, came out the following error when terminated, mitm.log is empty

Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

mitmweb mitmweb -s mitm_anatomy.py 2>&1 | tee /tmp/mitm.log only the stderr could be saved to mitm.log mitmweb mitmweb -s mitm_anatomy.py 2>&1 > /tmp/mitm.log did save both stderr and stdout to mitm.log but only flushed to the file after terminated. I needed the real-time log for tail -F to work with another program.

Had to do these to get what I wanted

import mitmproxy
from mitmproxy import ctx
import re

class AddHeader:
    def __init__(self):
        self.num = 0
        self.f = None
        
    def load(self, loader: mitmproxy.addonmanager.Loader):
        self.f = open('/tmp/mitm.log', 'w')
    
    def done(self): 
        self.f.close()
        
    def requestheaders(self, flow: mitmproxy.http.HTTPFlow):
        test_url = flow.request.pretty_url
        if re.search(r'https?:\/\/api4.my-ip.io\/ip[?&]pp=', test_url):
            #ctx.log(f'--- {flow.request.pretty_url}')
            print(f'--- {flow.request.pretty_url}')
            self.f.write(f'--- {flow.request.pretty_url}')
            self.f.flush()
        elif re.search(r'https?:\/\/api4.my-ip.io\/ip', test_url):
            #ctx.log(f'000 {flow.request.pretty_url}')
            print(f'000 {flow.request.pretty_url}')
            self.f.write(f'000 {flow.request.pretty_url}')
            self.f.flush()
            flow.kill()
                    
    def responseheaders(self, flow: mitmproxy.http.HTTPFlow):
        print(f'{flow.response.status_code} {flow.request.pretty_url}')
        self.f.write(f'{flow.response.status_code} {flow.request.pretty_url}')
        self.f.flush()

addons = [
   AddHeader()
]