RuntimeError when trying to mock httpx Response with async stream

When I try to mock a httpx Response with an async stream using respx, I get the following error:

RuntimeError: Attempted to call a sync iterator on an async stream.

I am running python 3.7.9, pytest 6.2.1, pytest-asyncio 0.14.0, httpx 0.16.1, respx 0.16.3

To replicate:

test_httpx.py

import httpx
import pytest
import respx


@respx.mock
@pytest.mark.asyncio
async def test_async_stream(tmpdir):
    dl_url = 'http://MockNotRealURL/download/?token=IAmAToken'

    img_headers = {
        'Content-Type': 'image/tiff',
        'Content-Length': '6',
        'Content-Disposition': 'attachment; filename="img.tif"'
    }

    async def _stream_img():
        yield b'bld'
        yield b'dce'

    mock_resp = httpx.Response(200, stream=_stream_img(), headers=img_headers)
    respx.get(dl_url).return_value = mock_resp

    filename = tmpdir / 'img.tif'

    async with httpx.AsyncClient() as cl:
        async with cl.stream('GET', dl_url) as resp:
            with open(filename, 'wb') as fp:
                async for chunk in resp.aiter_bytes():
                    fp.write(chunk)

Full traceback

>           async with cl.stream('GET', dl_url) as resp:

test_httpx.py:28: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_client.py:1835: in __aenter__
    stream=True,
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_client.py:1411: in send
    history=[],
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_client.py:1448: in _send_handling_auth
    history=history,
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_client.py:1476: in _send_handling_redirects
    response = await self._send_single_request(request, timeout)
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_client.py:1507: in _send_single_request
    ext={"timeout": timeout.as_dict()},
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/respx/mocks.py:137: in amock
    response = cls._send(request, instance=self, target_spec=spec, **kwargs)
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/respx/mocks.py:155: in _send
    httpx_response = cls.handler(httpx_request)
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/respx/mocks.py:86: in handler
    httpx_response = router.handler(httpx_request)
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/respx/router.py:270: in handler
    response = self.resolve(request)
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/respx/router.py:243: in resolve
    route, mock = self.match(request)
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/respx/router.py:231: in match
    response = prospect.match(request)
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/respx/models.py:394: in match
    result = self.resolve(request, **context)
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/respx/models.py:370: in resolve
    result = clone_response(result, request)
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/respx/models.py:62: in clone_response
    response.read()
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_models.py:1162: in read
    self._content = b"".join(self.iter_bytes())
../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_models.py:1175: in iter_bytes
    for chunk in self.iter_raw():
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Response [200 OK]>

    def iter_raw(self) -> typing.Iterator[bytes]:
        """
        A byte-iterator over the raw response content.
        """
        if self.is_stream_consumed:
            raise StreamConsumed()
        if self.is_closed:
            raise ResponseClosed()
        if not isinstance(self.stream, typing.Iterable):
>           raise RuntimeError("Attempted to call a sync iterator on an async stream.")
E           RuntimeError: Attempted to call a sync iterator on an async stream.

../../.pyenv/versions/3.7.9/envs/planet-client-python-3.7.9/lib/python3.7/site-packages/httpx/_models.py:1209: RuntimeError

[jreiberkyle]

@lundberg that fixes it!

[jreiberkyle]

The issue is actually caused in respx.models.clone_response calling response.read() when the stream is an AsyncIterator. For an async stream, response.aread() should be called.

Ultimately, looking into clone_response(), it looks like it is calling response.read() to populate response.content. I’ve tried adjusting clone_response() to populate response.content directly, but it is causing some tests to fail so I’m not there yet…

def clone_response(response: httpx.Response, request: httpx.Request) -> httpx.Response:
    """
    Clones a httpx Response for given request.
    """

    # new
    try:
        content = response.content
    except httpx.ResponseNotRead:
        content = None

    response = httpx.Response(
        response.status_code,
        headers=response.headers,
        content=content,  # new
        stream=response.stream,
        request=request,
        ext=dict(response.ext),
    )

    # removed: response.read()
    return response