Missing support for encoding x-fb-dz

Problem Description

A clear and concise description of what the bug is.

Hi ! I’m using mitmproxy to intercept Facebook traffic from their Android app to https://graph.facebook.com/graphql, and they seem to have recently introduced a new encoding called x-fb-dz according to the content-type header. This encoding is unknown to mitmproxy and causes the following error, which prevents response bodies from being decoded.

Addon error: Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/mitmproxy/net/http/encoding.py", line 65, in decode
    decoded = custom_decode[encoding](encoded)
KeyError: 'x-fb-dz'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/mitmproxy/net/http/encoding.py", line 67, in decode
    decoded = codecs.decode(encoded, encoding, errors)  # type: ignore
LookupError: unknown encoding: x-fb-dz

This encoding might be zstd using a dictionary trained on a sample of their responses, judging by the presence of a x-fb-dz-dict header, but I might be wrong as I don’t know much about encodings.

Are there plans to support this encoding or is this out of scope for mitmproxy ?

Steps to reproduce the behavior:

1. Set up a researcher profile to allow user installed certificates on Android device
2. Intercept a POST request to https://graph.facebook.com/graphql with header x-fb-friendly-name == fresh_feed_new_data_fetch
3. Observe that body cannot be decoded and that content-type == x-fb-dz

System Information

Paste the output of “mitmproxy --version” here.

Mitmproxy: 6.0.2
Python:    3.9.1
OpenSSL:   OpenSSL 1.1.1i  8 Dec 2020
Platform:  Linux-5.10.6-arch1-1-x86_64-with-glibc2.32

[mame82]

Ended up with the same problem for traffic directed to graph.facebook.com/graphql and agree that there is a high probability, that zstd with a custom dictionary is used. In fact the dictionary to use is indicated by the requesting client within the accept- encoding header.

Example (server gets advised to use dictionary 1 for zstd):

accept-encoding:      x-fb-dz;d=1, zstd, gzip, deflate

Server response indicates that dictionary 1 was used for compression:

x-fb-dz-dict:         1
content-encoding:     x-fb-dz

I found no indication, whatsoever, that the dictionary gets transmitted, it is likely hardcoded on the client-side (or there is a Facebook specific API request to fetch it).

Obviously, it can not be the task of mitmproxy, to embed suiting dictionaries for all possible hosts using zstd compression. Yet, in the case of FB graphql traffic, zstd compression could safely be avoided, by removing it from the sequence of allowed compressions in the accept-encoding header.

Gonna leave this here, in case others struggle with the same problem:

Snippet of mitmproxy request hook which modifies accept-encoding headers to avoid zstd compression for Facebook GraphQL:

    def replace_x_fb_dz(self, flow: http.HTTPFlow):
        ae: Union[str, None] = flow.request.headers.get("accept-encoding")
        if ae is not None and ae.startswith("x-fb-dz"):
            # remove zstd compression with unknown dict in AE-headers like "x-fb-dz;d=1, zstd, gzip, deflate"
            flow.request.headers["accept-encoding"] = "x-fb-dz;gzip, deflate"

    def request(self, flow: http.HTTPFlow):
        self.replace_x_fb_dz(flow=flow)  # replace zstd encoding with unknown dict for FB traffic

[mhils]

Thanks @mame82! This is interesting work. The easiest approach here is to just set mitmproxy’s anticomp option to remove accept-encoding headers entirely.

Putting my tinfoil hat hon, this of course makes your client stick out a bit and you kind of need to trust Facebook not to alter their behavior when they know that traffic is being watched. Practically speaking I think this is fine - I don’t see much of a new problem here, a malicious adversary could already do that client-side based on the interception certificate (or server-side based on TLS fingerprinting).