[Core] Specify the serializer for a certain object

Describe your feature request

The default serializer has many issues with certain objects. One case is the pydantic model. We can specify a flag __ray_serializer__ in these object types to enforce ray using pickle as the default serializer instead. This avoids changing other functions like ray.put, ray.remote and xxxx.remote(...). This could also cover our major cases and is super flexible.

One example could be:

from pydantic import BaseModel, validator

class UserModel(BaseModel):
    __ray_serializer__ = 'pickle'
    name: str
    username: str
    password1: str
    password2: str

    @validator('name')
    def name_must_contain_space(cls, v):
        if ' ' not in v:
            raise ValueError('must contain a space')
        return v.title()

    @validator('password2')
    def passwords_match(cls, v, values, **kwargs):
        if 'password1' in values and v != values['password1']:
            raise ValueError('passwords do not match')
        return v

    @validator('username')
    def username_alphanumeric(cls, v):
        assert v.isalnum(), 'must be alphanumeric'
        return v

[suquark]

yeah, a trivial solution is to overload __reduce__ with pickle.dumps, so cloudpickle then serialize the pickled content again.

def __reduce__(self):
   return pickle.loads, (pickle.dumps(self),)

[simon-mo]

I think @pcmoritz has a great idea here:

Hey Simon, I think you can circumvent the requirement to pickle the class for reduce if you do something like this:

In [10]: def unpickle(*args):
    ...:     return # ...
    ...: class Foo(object):
    ...:   def __init__(self, file_name = 'file.txt'):
    ...:     self.file_name = file_name
    ...:     self.f = open(self.file_name, 'w')
    ...: 
    ...:   def __reduce__(self):
    ...:     return (unpickle, (self.file_name, ))

I.e. instead of putting the class as the first argument for the return value of reduce, put a custom function.

Now I just need to define unparse as:

def unparse(*args):
    from package import Foo
    return Foo(*args)