HTTPX-based TestClient?

Hi folks,

Currently, the TestClient is built as a subclass of requests.Session.

What would be thoughts on migrating to HTTPX?

• Is this a good idea at all?
• Should the TestClient be built around HTTPX, or should it eventually be replaced by HTTPX and its ASGI support?

As of today, there are definitely blockers to a full migration. An important one is that HTTPX doesn’t support WebSocket yet (see https://github.com/encode/httpx/issues/304), while TestClient does.

Happy to discuss! cc @tiangolo @gvbgduh @tomchristie

[Congee]

For those who use FastAPI and would like to use httpx instead of requests for testing, I have stolen and adapted some code from our awesome @tomchristie

#!/usr/bin/env python3

# Took credit from @tomchristie
# https://github.com/encode/hostedapi/blob/master/tests/conftest.py

import asyncio

import pytest
from starlette.config import environ

from app.main import app
from tests.client import TestClient

# This sets `os.environ`, but provides some additional protection.
# If we placed it below the application import, it would raise an error
# informing us that 'TESTING' had already been read from the environment.
environ["TESTING"] = "True"


# According to pytest-asyncio README, we should create a new event loop fixture
# to avoid using the default one.
# No. I have to use the default event loop. Because I do not know why our event
# loop is always closed early.
# https://github.com/pytest-dev/pytest-asyncio/blob/86cd9a6fd2/pytest_asyncio/plugin.py#L169-L174
@pytest.fixture(scope='module')
def event_loop():
    # loop = asyncio.get_event_loop_policy().new_event_loop()
    # asyncio.set_event_loop(loop)
    yield asyncio.get_event_loop()
    # assert loop.is_running()


@pytest.fixture(scope='module')
async def client():
    """
    When using the 'client' fixture in test cases, we'll get full database
    rollbacks between test cases:

    async def test_homepage(client):
        url = app.url_path_for('homepage')
        response = await client.get(url)
        assert response.status_code == 200
    """
    async with TestClient(app=app) as client:
        yield client

#!/usr/bin/env python3

import asyncio
import typing
from types import TracebackType

from httpx import Client
from httpx.dispatch.asgi import ASGIDispatch
from starlette.types import ASGIApp


class TestClient(Client):
    __test__ = False
    token: str = None

    def __init__(self, app: ASGIApp, *args, **kwargs) -> None:
        self.app = app
        super().__init__(
            dispatch=ASGIDispatch(app=app),
            base_url='http://testserver',
            *args,
            **kwargs
        )


    async def ensure_authed(self):
        if self.token is None:  # TODO: expiry
            req = self.build_request(
                'POST',
                f'{self.base_url}/token',
                json={
                    "grant_type": "client_credentials",
                    "client_id": "dddaffc6-c786-48b7-8e3c-9f8472119939",
                    "client_secret": "password"
                }
            )
            resp = await self.send(req)
            assert resp.status_code == 200
            self.token = resp.json()['access_token']

        if 'Authorization' not in self.headers:
            self.headers = self.merge_headers({
                'Authorization': f'Bearer {self.token}'
            })


    async def request(self, *args, **kwargs):
        await self.ensure_authed()
        return await super().request(*args, **kwargs)


    async def lifespan(self) -> None:
        """ https://asgi.readthedocs.io/en/latest/specs/lifespan.html """
        scope = {'type': 'lifespan'}
        await self.app(scope, self.recv_queue.get, self.send_queue.put)
        await self.send_queue.put(None)


    async def wait_startup(self) -> None:
        await self.recv_queue.put({'type': 'lifespan.startup'})
        message = await self.send_queue.get()
        assert message['type'] in {
            'lifespan.startup.complete',
            'lifespan.startup.failed',
        }
        if message['type'] == 'lifespan.startup.failed':
            message = await self.send_queue.get()
            if message is None:
                self.task.result()


    async def wait_shutdown(self) -> None:
        await self.recv_queue.put({'type': 'lifespan.shutdown'})
        message = await self.send_queue.get()
        if message is None:
            self.task.result()

        assert message['type'] == 'lifespan.shutdown.complete'
        await self.task


    async def __aenter__(self) -> Client:
        self.send_queue = asyncio.Queue()
        self.recv_queue = asyncio.Queue()
        self.task: asyncio.Task = asyncio.create_task(self.lifespan())
        await self.wait_startup()
        return self


    async def __aexit__(
        self,
        exc_type: typing.Type[BaseException] = None,
        exc_value: BaseException = None,
        traceback: TracebackType = None,
    ) -> None:
        await self.wait_shutdown()
        await self.close()

[tiangolo]

I’m pretty sure that’s part of the initial objective, to move to HTTPX instead of Requests.

Given it’s basically the same interface as Requests, I don’t think there would be any problem with it for users/developers.

I’m personally happy with it. 🚀

Of course, after solving the blockers, but I think it’s a good idea. It should also give more flexibility and control on how to do stuff, how to extend, etc.

I guess we should support the TestClient, probably as a wrapper, at least for some time, to not break existing implementations, even if the decision was to drop the TestClient completely to fully move to HTTPX and and mark the TestClient as deprecated in favor of HTTPX.

I also think we should start adding “how to use HTTPX” to docs, as it’s the perfect match for async frameworks when communicating to third parties. But that’s another topic.

@dmontagu you have been helping a looot with FastAPI recently ( 🙇‍♂️ 👏 ), do you see any obvious drawbacks to this?