Latest dill release raises exception

Describe the bug

As reported by @sgugger, latest dill release is breaking things with Datasets.

______________ ExamplesTests.test_run_speech_recognition_seq2seq _______________


self = <multiprocess.pool.ApplyResult object at 0x7fa5981a1cd0>, timeout = None

    def get(self, timeout=None):
        self.wait(timeout)
        if not self.ready():
            raise TimeoutError
        if self._success:
            return self._value
        else:
>           raise self._value
E           TypeError: '>' not supported between instances of 'NoneType' and 'float'

[anivegesana]

I should note that because Python 3.6 and older are now deprecated and Python 3.7 has insertion order dictionaries, the globals in dill will have a deterministic order, just not sorted. I would still keep it sorted like you have it to help with stability (for example, if someone reorders variables in a file, then sorting the globals would not invalidate the cache.)

It seems that the order is not quite deterministic in IPython. Huggingface datasets seems to do well in Jupyter regardless, so it is not a good idea to remove the sorting. uqfoundation/dill#19

[albertvillanova]

Thanks a lot for your investigation @anivegesana.

Yes, we copied-pasted the old save_function function from dill, just adding a line to make deterministic the order of global variables globs.

However, this function has changed a lot from version 0.3.5, after your PR (thank you for the fix in recursiveness, indeed):

• uqfoundation/dill#443

We have to address this change.

If finally your PR to sort global variables is merged into dill 0.3.6, that will make our life easier, as the tweak will no longer be necessary. 😉

I have included a regression test so that we are sure future releases of dill do not break datasets:

• #4385