Add support for non-HTTP flows to the UI.

We want to support other protocols than basic HTTP in the UI. In particular, we want to display HTTP2-specific content, WebSockets and raw tcp streams at some point.

The basic foundations for this are laid out: We have a generic Flow object (mitmproxy.models.flow) and a HTTPFlow that subclasses it (libmproxy.models.http). For TCP, we would introduce a TCPFlow object that stores a list of messages objects from client and server (in comparison, the HTTPFlow stores a distinct request and response). libmproxy.protocol.tcp would then create one TCPFlow per connection.

Each connection thread passes Flow objects to the FlowMaster (handle_request, handle_response, …), which stores them in a FlowStore (see libmproxy.flow). The UI renders the store contents.

The high level implementation steps are as follows:

• Set up TCPFlow class
• Send TCPFlows to the FlowMaster (e.g. handle_new_tcp_flow, handle_update_tcp_flow)
• Add printing to mitmdump.
• Make mitmdump -w and mitmdump -r work.
• Display TCP flows in the UI.
• Display WebSocket flows in the UI.

[mhils]

@Jessonsotoventura @nikitastupin and @jnsnow: Here’s what I will do: I’ll cherry-pick things from both your PRs today and get a first basis (Step 1 in https://github.com/mitmproxy/mitmproxy/issues/1020#issuecomment-609906840) merged into mitmproxy as soon as possible. We can then iterate from there on, @Jessonsotoventura can spray his color highlighting magic, etc. 😃 🚀

[mhils]

I absolutely agree! If your or anyone else wants to tackle this, I’d be more than happy to support that effort. Generally speaking, I think we can split this work into multiple parts which can be isolated PRs:

1. Display TCP flows in the flow list and provide a simple detail view. See https://github.com/mitmproxy/mitmproxy/pull/3680#pullrequestreview-317233135 for some thoughts on the UI. This is the most fundamental part, the adventure starts with the steps outlined in https://github.com/mitmproxy/mitmproxy/issues/1020#issuecomment-408521233. The hard part here is not to get sidetracked and start with a minimal PR that can be merged quickly.
2. Integrate our contentviews so that messages can be pretty-printed.
3. Implement interception and modification for individual messages.
4. Implement flow-level modification (reorder messages, merge/split messages, …) and replay functionality.

There is also an early prototype by @Jessonsotoventura in #3680 which hasn’t been mentioned here yet. Maybe someone takes some inspiration from that or he finds some time to pick this up again. In any case, please feel free to reach out on Slack if we can help. 😃