[BUG] FastAPI doesn't return router result list when using asyncpg

Describe the bug

I took the weekend to study a little the fastapi project, to try creating a full async HTTP request lifecycle (including IO with database). I had success (there are lots of things to improve and understand better) but I found a strange behavior when using postgres with asyncpg. When trying to fetch users from the endpoint [1], a server error is returned, and lots of validations errors are shown in the console. I think that is some strange behavior with asyncpg.Record, because in this point [2] the variable response_content has a list of records representing the result of the query executed. I tried changing the backend to postgres+aiopg and this problem doesn’t happen. With this backend, the endpoint works as expected.

1 - http://localhost:8000/users/?q=User%20Name&limit=1&offset=0 2 - https://github.com/tiangolo/fastapi/blob/master/fastapi/routing.py#L116

To Reproduce

Create the main app

app = FastAPI()
app.include_router(routers.router)

db = databases.Database(settings.DATABASE_URL)


@app.middleware("http")
async def db_session_middleware(request: Request, call_next):
    request.state.db = db
    response = await call_next(request)
    return response


@app.on_event("startup")
async def startup():
    await db.connect()


@app.on_event("shutdown")
async def shutdown():
    await db.disconnect()

Create a router to fetch a query with many results and defining a response with typing.List[Model]

router = APIRouter()


@router.get(
    "/users/", 
    description="Endpoint to search users", 
    response_model=typing.List[User],
    dependencies=[Depends(Authentication)],
)
async def users(request: Request, p: Pagination = Depends()):
    query = select([user_table]).where(user_table.c.name == p.q).limit(p.limit).offset(p.offset)
    result = await request.state.db.fetch_all(query=query)
    
    return result

Create a model to be tied with the router

class User(BaseModel):
    id: UUID = ""
    address: Address = None
    email: EmailStr
    is_active: bool
    name: str = Field(..., min_length=8, max_length=64, description="The name that represents the user")
    permissions: typing.List[str] = []

    class Config:
        orm_mode = True

Database URL

DATABASE_URL=postgresql://users_api:users_api@localhost/users_api

Error

INFO:     127.0.0.1:56901 - "GET /users/?q=User%20Name&limit=1&offset=0 HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 385, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/fastapi/applications.py", line 149, in __call__
    await super().__call__(scope, receive, send)
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/starlette/applications.py", line 102, in __call__
    await self.middleware_stack(scope, receive, send)
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/starlette/middleware/base.py", line 25, in __call__
    response = await self.dispatch_func(request, self.call_next)
  File "./app/main.py", line 20, in db_session_middleware
    response = await call_next(request)
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/starlette/middleware/base.py", line 45, in call_next
    task.result()
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/starlette/middleware/base.py", line 38, in coro
    await self.app(scope, receive, send)
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/starlette/routing.py", line 550, in __call__
    await route.handle(scope, receive, send)
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/fastapi/routing.py", line 205, in app
    response_data = await serialize_response(
  File ".pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/fastapi/routing.py", line 126, in serialize_response
    raise ValidationError(errors, field.type_)
pydantic.error_wrappers.ValidationError: 3 validation errors for User
response -> 0 -> email
  field required (type=value_error.missing)
response -> 0 -> is_active
  field required (type=value_error.missing)
response -> 0 -> name

Pdb

> /.pyenv/versions/3.8.2/envs/sample-fast-api/lib/python3.8/site-packages/fastapi/routing.py(117)serialize_response()
-> value, errors_ = field.validate(response_content, {}, loc=("response",))
(Pdb) ll
 94     async def serialize_response(
 95         *,
 96         field: ModelField = None,
 97         response_content: Any,
 98         include: Union[SetIntStr, DictIntStrAny] = None,
 99         exclude: Union[SetIntStr, DictIntStrAny] = set(),
100         by_alias: bool = True,
101         exclude_unset: bool = False,
102         exclude_defaults: bool = False,
103         exclude_none: bool = False,
104         is_coroutine: bool = True,
105     ) -> Any:
106         if field:
107             errors = []
108             response_content = _prepare_response_content(
109                 response_content,
110                 by_alias=by_alias,
111                 exclude_unset=exclude_unset,
112                 exclude_defaults=exclude_defaults,
113                 exclude_none=exclude_none,
114             )
115             if is_coroutine:
116                 breakpoint()
117  ->             value, errors_ = field.validate(response_content, {}, loc=("response",))
118             else:
119                 value, errors_ = await run_in_threadpool(
120                     field.validate, response_content, {}, loc=("response",)
121                 )
122             if isinstance(errors_, ErrorWrapper):
123                 errors.append(errors_)
124             elif isinstance(errors_, list):
125                 errors.extend(errors_)
126             if errors:
127                 raise ValidationError(errors, field.type_)
128             return jsonable_encoder(
129                 value,
130                 include=include,
131                 exclude=exclude,
132                 by_alias=by_alias,
133                 exclude_unset=exclude_unset,
134                 exclude_defaults=exclude_defaults,
135                 exclude_none=exclude_none,
136             )
137         else:
138             return jsonable_encoder(response_content)
(Pdb) response_content
[<databases.backends.postgres.Record object at 0x10c3d5430>]
(Pdb) response_content[0]._row
<Record id=UUID('e108216a-01fe-4974-b1c1-e4858be9bb91') address_id=UUID('12f6fb53-821d-48c2-b429-154694155e79') email='[e-mail]' is_active=True name='User Name' password='de3f039bb1b804e1f8b1641beeeff1e2f2f96d100d79a354fe0e52780dda43173d76f71eb935c82d5a4e748d790d4fb7e274ef81833f906a47f2c33f612801dea7c7b4db55ceae37ddbea56456b8b561c3291d1e195227003aa9b0c80455d53f' permissions=['create', 'update', 'delete']>

Changing to postgres+aiopg the response as showed bellow and all results are returned as expected

INFO:     127.0.0.1:57000 - "GET /users/?q=User%20Name&limit=1&offset=0 HTTP/1.1" 200 OK

Environment

• OS: macOS
• FastAPI Version 0.54.1
• Python version 3.8.2
• SQLAlchemy 1.3.16
• asyncpg 0.20.1
• databases 0.3.2
• email-validator 1.1.0
• pydantic 1.5.1
• starlette 0.13.2
• uvicorn 0.11.5
• uvloop 0.14.0

[cbows]

I can confirm this issue. If I use databases with the postgres+asyncpg backend, I get a lot of field required (type=value_error.missing) errors. Using the same query, and instantiating the model manually in ipython via Model(**record) works as expected. Without response_model, I can confirm that all relevant fields exist, are of correct type and are returned to the client, but adding response_model=typing.List[Model] raises said errors.

[fvlima]

So, I will close this issue since that is not a problem here. Maybe it can be handled by the pydantic project, but I think this is a topic for another discussion. If we can just override this behavior with AttrDict, there is a way to resolve this problem bu now.