Feature Request: 3rd party non-JSON serialization/deserialization

Hi, author of pydantic-yaml here. I have no idea about anything Rust-related, unfortunately, but hopefully this feature request will make sense in Python land.

I’m going off this slide in this presentation by @samuelcolvin, specifically:

We could add support for other formats (e.g. yaml, toml) the only side affect would be bigger binaries.

Here’s a relevant discussion about “3rd party” deserialization from v1: https://github.com/samuelcolvin/pydantic/discussions/3025

It would be great if pydantic-core were built in a way where non-JSON formats could be added “on top” rather than necessarily being built into the core. I understand performance is a big question in this rewrite, so ideally these would be high-level interfaces that can be hacked in Python (or implemented in Rust/etc. for better performance).

From the examples available already, it’s possible that such a feature could be quite simple on the pydantic-core side - the 3rd party would create their own function a-la validate_json, possibly just calling validate_python. However, care would be needed on how format-specific details are sent between pydantic and the implementation. In V1 this is done with the Config class and special json_encoder/decoder attributes, which have been a pain to re-implement for YAML properly (without way too much hackery).

Ideally for V2, this would be something more easily addable and configurable. The alternative would be to just implement TOML, YAML etc. directly in the binary (and I wouldn’t have to keep supporting my project, ha!)

Thanks again for Pydantic!

[samuelcolvin]

Thanks for the question.

I think this is just calling validate_python, or indeed initializing the pydantic model as I guess you do now.

No changes should be required to pydantic-core to allow this.

I want to add support for line numbers in errors, but that requires a new rust JSON parser, so won’t be added until v2.1 or later.

[samuelcolvin]

This sounds very hard to do in a reliable cross-platform way. Given the problems we already experience (at the scale of pydantic or watchfiles) with very subtle OS differences and wheels. I’m very unwilling to enter into this mess.

You’re effectively proposing another way link libraries that side steps both crates and python imports, are there any other examples of python libraries that use DLL/ share libraries to access functionality in other packages without using the python runtime?

(Perhaps worth remembering that I’ll probably be maintaining pydantic for the next decade at least, one “clever” idea proposed now which relies on shaky cross-platform behaviour, could cost me 100s of hours of support over the years - hence my caution)

I real question is how much faster would this be than my proposal above?

To proceed with the conversation, we need to benchmark that. Really someone need to build both solutions (and any 3rd solutions proposed) and see how they perform.

@PrettyWood @tiangolo do you have any feedback on any of this?