Multipart form handling weird behavior

Background

Hey everyone!

I’m writing an app that processes multipart form with file upload. When I serve it with uvicorn and run cURL against the endpoint I get an error curl: (56) Illegal or missing hexadecimal sequence in chunked-encoding. Data from backend reaches cURL, but it returns an odd error.

Looking into ascii-trace I can see that data chunk from backend mystically comes to cURL without length line which should be at the beginning of chunk.

<= Recv data, 20 bytes (0x14)
0000: Hello, world!
000f: 0
0012:

However, dtrace and Wireshark showed me that chunk data was sent from uvicorn with length as it should. Other interesting fact is that I get no cURL error if serving app with daphne(daphne -p 8000 app:App).

What am I missing? 🤔

Environment

1. Mac OS X
2. uvicorn==0.3.21
3. Python 3.7.0 (with pipenv)
4. uvloop / asyncio gives same result

Steps to reproduce

1. Create snippet:

class App():
    def __init__(self, scope):
        self.scope = scope

    async def __call__(self, receive, send):
        while True:
            message = await receive()
            print('(%s) message from  recv is %r' % (id(self),message))
            if message["type"] == "http.disconnect":
                return
            else:
                if not message.get("more_body"):
                    await send({
                        'type': 'http.response.start',
                        'status': 200,
                        'headers': [[b'content-type', b'text/plain']]
                    })
                    await send({
                        'type': 'http.response.body',
                        'body': b'Hello, world!',
                        'more_body': False
                    })

1. Run http server

python uvicorn app:App

1. Query with cURL

curl  --trace-ascii curl.trace-uvicorn \
            --form file=@foo \
            --form file2=@foo \
            http://localhost:8000/api/upload
curl: (56) Illegal or missing hexadecimal sequence in chunked-encoding

1. Check what’s in trace

$ cat curl.trace-uvicorn
== Info:   Trying ::1...
== Info: TCP_NODELAY set
== Info: Connection failed
== Info: connect to ::1 port 8000 failed: Connection refused
== Info:   Trying fe80::1...
== Info: TCP_NODELAY set
== Info: Connection failed
== Info: connect to fe80::1 port 8000 failed: Connection refused
== Info:   Trying 127.0.0.1...
== Info: TCP_NODELAY set
== Info: Connected to localhost (127.0.0.1) port 8000 (#0)
=> Send header, 218 bytes (0xda)
0000: POST /api/upload HTTP/1.1
001b: Host: localhost:8000
0031: User-Agent: curl/7.54.0
004a: Accept: */*
0057: Content-Length: 361
006c: Expect: 100-continue
0082: Content-Type: multipart/form-data; boundary=--------------------
00c2: ----68de9f32447ad9b5
00d8:
<= Recv header, 23 bytes (0x17)
0000: HTTP/1.1 100 Continue
== Info: Done waiting for 100-continue
=> Send data, 147 bytes (0x93)
0000: --------------------------68de9f32447ad9b5
002c: Content-Disposition: form-data; name="file"; filename="foo"
0069: Content-Type: application/octet-stream
0091:
=> Send data, 8 bytes (0x8)
0000: content.
=> Send data, 150 bytes (0x96)
0000:
0002: --------------------------68de9f32447ad9b5
002e: Content-Disposition: form-data; name="file2"; filename="foo"
006c: Content-Type: application/octet-stream
0094:
=> Send data, 8 bytes (0x8)
0000: content.
=> Send data, 48 bytes (0x30)
0000:
0002: --------------------------68de9f32447ad9b5--
<= Recv header, 17 bytes (0x11)
0000: HTTP/1.1 200 OK
<= Recv header, 17 bytes (0x11)
0000: server: uvicorn
<= Recv header, 37 bytes (0x25)
0000: date: Tue, 20 Nov 2018 04:21:02 GMT
<= Recv header, 26 bytes (0x1a)
0000: content-type: text/plain
<= Recv header, 28 bytes (0x1c)
0000: transfer-encoding: chunked
<= Recv data, 20 bytes (0x14)
0000: Hello, world!
000f: 0
0012:
== Info: Illegal or missing hexadecimal sequence in chunked-encoding
== Info: stopped the pause stream!
== Info: Closing connection 0

[tomchristie]

Released as 0.3.26

[tomchristie]

Compared the bytewise output of h11 vs httptools, and deteremined we were writing 100 continue in an invalid way. Required an additional line ending pair.