Support Base64 Encoding LargeBinary Field

Is your feature request related to a problem? Please describe.

In a Model such as:

class BaseMessageModel(Model):
    class Meta:
        metadata = metadata
        database = database

    data: bytes = LargeBinary(max_length=1000)

One of the challenges that comes up is actually serializing this binary data effectively. If there are any UTF-8 incompatible bytes:

Traceback (most recent call last):
  File "venv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 398, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "venv/lib/python3.9/site-packages/fastapi/applications.py", line 199, in __call__
    await super().__call__(scope, receive, send)
  File "venv/lib/python3.9/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "venv/lib/python3.9/site-packages/starlette/middleware/cors.py", line 78, in __call__
    await self.app(scope, receive, send)
  File "venv/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "venv/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "venv/lib/python3.9/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "venv/lib/python3.9/site-packages/starlette/routing.py", line 376, in handle
    await self.app(scope, receive, send)
  File "venv/lib/python3.9/site-packages/fastapi/applications.py", line 199, in __call__
    await super().__call__(scope, receive, send)
  File "venv/lib/python3.9/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "venv/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "venv/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "venv/lib/python3.9/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "venv/lib/python3.9/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "venv/lib/python3.9/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "venv/lib/python3.9/site-packages/fastapi/routing.py", line 209, in app
    response_data = await serialize_response(
  File "venv/lib/python3.9/site-packages/fastapi/routing.py", line 127, in serialize_response
    return jsonable_encoder(
  File "venv/lib/python3.9/site-packages/fastapi/encoders.py", line 104, in jsonable_encoder
    jsonable_encoder(
  File "venv/lib/python3.9/site-packages/fastapi/encoders.py", line 57, in jsonable_encoder
    return jsonable_encoder(
  File "venv/lib/python3.9/site-packages/fastapi/encoders.py", line 90, in jsonable_encoder
    encoded_value = jsonable_encoder(
  File "venv/lib/python3.9/site-packages/fastapi/encoders.py", line 127, in jsonable_encoder
    return ENCODERS_BY_TYPE[type(obj)](obj)
  File "pydantic/json.py", line 43, in pydantic.json.lambda
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 1: invalid start byte

This makes sense since you can’t send arbitrary bytes through a fundamentally text-based (UTF-8) JSON representation.

Describe the solution you’d like

class BaseMessageModel(Model):
    class Meta:
        metadata = metadata
        database = database

    data: str = LargeBinary(max_length=1000, represent_as_base64=True)

And internally this would call something like:

    @validator("data")
    def convert_to_base64_str(cls, v: bytes) -> str:
        return base64.b64encode(v).decode()

Edit: It looks like FastAPI doesn’t use OpenAPI 3.1 yet (it is newly released), so probably the older 3.0 format should be used:

https://github.com/OAI/OpenAPI-Specification/blob/fbe62006211838a8bb7bf2433a1d15f1a5838a03/versions/3.0.1.md#considerations-for-file-uploads

# content transferred with base64 encoding
schema:
  type: string
  format: base64

Future Considerations

In OpenAPI 3.1 (not yet supported by FastAPI): https://github.com/OAI/OpenAPI-Specification/issues/1547#issuecomment-595497718

type: string
contentEncoding: base64

Describe alternatives you’ve considered

Ideally Pydantic would support this serialization upstream: https://github.com/samuelcolvin/pydantic/issues/692 but it’s been that issue has been open for 2 years.

[collerek]

That’s what you get when one simple change requires major refactor of some aspects 🤣 At least we gained some performance speed (and now it’s more elegant, never liked overly complicated __getattribute__).

Tracked in a new issue so now I don’t miss it again 😉

[collerek]

Right now it’s not happening, a possible use case would be to convert string to bytes on init/set and store it as bytes. So by default it’s bytes -> bytes, but if you provide a string it will be str -> convert -> bytes and later on fetch it will be bytes (so one way convert).