HTTP/2 download speeds / flow control settings

Which having an initial look at HTTP/3, @jlaine dug out that we seem to have signiifcantly slow uploading right now.

First thing to do would be to investigate and replicate, by eg. compare and contrast a simple upload from httpx vs requests/urllib3 - does it replicate trivially and reliably? This really shouldn’t be an issue for us so will need some looking at.

Initial thoughts on where issues could be:

• We’re sending too-small chunks, and always waiting for the network to fully drain on each chunk is negatively impacting performance.
• We’re sending too-small chunks, and ending up with largish computation overhead relative to data transfer as a result.
• We’ve got some big unseen overhead in the sync-to-async marshalling.
• Something else?

[jlaine]

The server mentioned (quic.aiortc.org) above is serving HTTP/1.1 and HTTP/2 using nginx talking (over HTTP/1.1) to uvicorn.

Using cURL over HTTP/1.1 or HTTP/2 against the same server as above both give me times which are consistent with requests and httpx over HTTP/1.1 (i.e. around 6s). httpx over HTTP/2 really seems to be the outlier.

curl --http1.1 https://quic.aiortc.org/50000000 curl --http2 https://quic.aiortc.org/50000000

The httpbin test you gave above isn’t super exciting, it only transfers ~100kB of data vs 50MB in mine 😃

[tomchristie]

We’ve got a much cleaner HTTP/2 implementation now, and this still replicates.

I strongly suspect that it’s a flow control issue. Perhaps something like our allowable window sizes being set too low by default, and having to send too many window updates.

In any case, we really need someone to try to dig into a proper comparision against some other tool (curl or a browser implementation) too see how our messaging differs.

I guess it’s also possible that the messaging is correct, but there’s a networking implementation issue (eg. perhaps we really need the writes to be on a different flow of control here.) I don’t particularly expect that’s the case, but it might be interesting to see if the issue persists on asyncio if we stop calling .drain() after every write?