allow objects to define their own tokenize

Recently, I found it convenient for objects to define their own tokenization. Basically, by checking if it has the attribute tokenize. Would this be something that could be supported by dask? If this has already been discussed, I apologize in advance. I searched very quickly.

Thanks!

example:

    def tokenize(*args, **kwargs):

        # now check if each argument/kwarg handle their own tokenization
        argslist = list()
        for arg in args:
            if hasattr(arg, 'tokenize'):
                argslist.append(arg.tokenize())
            else:
                argslist.append(arg)

        kwargsdict = dict()
        for key, val in kwargs.items():
            if hasattr(val, 'tokenize'):
                kwargsdict[key] = val.tokenize()
            else:
                kwargsdict[key] = val

        hsh = tokenize(*argslist, **kwargsdict, pure=True)
       
        return hsh

[IPetrik]

For those like me coming here after Oct 2017, the method __dask_tokenize__ was added. (http://docs.dask.org/en/latest/custom-collections.html#deterministic-hashing)

[jcrist]

maybe this should be added to the documentation?

I’m not 100% set on this being public api yet, but I wouldn’t expect it to change anytime soon so you should be fine relying on it.