Configuring specific token hashing algorithm

Would be nice to have a way to force a particular hashing algorithm with Dask via set_options. Stealing the code snippet below from @jcrist’s comment. This is needed for use cases where having reproducible hashes are necessary (e.g. caching, versioning, etc.).

dask.set_options(tokenize_hash='mmh3')  # use mmh3, error if not available
dask.set_options(tokenize_hash=None)   # Use best available hash, default behavior

[jcrist]

You’d make the hash_buffer function check some global option first, and fallback to the default if unset. This is a common pattern in dask, see e.g. https://github.com/dask/dask/blob/master/dask/base.py#L218. Note that currently the fast hashers are only used for things like numpy arrays, and not for all tokenize operations, so the option name might be dask.set_options(hash_buffer='mmh3') or something similar.

[jakirkham]

It remains interesting. Would describe it as medium to long range priority for me though.

Guess the strategy for solving this requires revisiting as this preceded the config revamp.