TypeError: an integer is required (got type bytes)

Since upgrading to python 3.8 I can’t access my old jobs’ submission pickle (error below).

The problem might be related to this issue or this one but I have no clue what it means.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-17-0248393e65dd> in <module>
     30         if state != "COMPLETED":
     31             continue
---> 32         row = job.submission().kwargs
     33         row["scores"] = job.result()
     34         row["exp_name"] = exp_name

~/dev/ext/submitit/submitit/core/core.py in submission(self)
    206             self.paths.submitted_pickle.exists()
    207         ), f"Cannot find job submission pickle: {self.paths.submitted_pickle}"
--> 208         return utils.DelayedSubmission.load(self.paths.submitted_pickle)
    209 
    210     def cancel_at_deletion(self, value: bool = True) -> "Job[R]":

~/dev/ext/submitit/submitit/core/utils.py in load(cls, filepath)
    133     @classmethod
    134     def load(cls: Type["DelayedSubmission"], filepath: Union[str, Path]) -> "DelayedSubmission":
--> 135         obj = pickle_load(filepath)
    136         # following assertion is relaxed compared to isinstance, to allow flexibility
    137         # (Eg: copying this class in a project to be able to have checkpointable jobs without adding submitit as dependency)

~/dev/ext/submitit/submitit/core/utils.py in pickle_load(filename)
    271     # this is used by cloudpickle as well
    272     with open(filename, "rb") as ifile:
--> 273         return pickle.load(ifile)
    274 
    275 

TypeError: an integer is required (got type bytes)

Repro: Start a job with python 3.7 and then try to access it in python 3.8. In python 3.7

Python 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import submitit
>>>
>>> def add(a, b):
...     return a + b
...
>>> executor = submitit.AutoExecutor(folder="log_test")
>>> executor.update_parameters(timeout_min=1, slurm_partition="dev")
>>> job = executor.submit(add, 5, 7)
>>> print(job.job_id)
33389760
>>> job.submission()
<submitit.core.utils.DelayedSubmission object at 0x7f42f5952bd0>

In python 3.8

Python 3.8.5 | packaged by conda-forge | (default, Jul 24 2020, 01:25:15)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import submitit
>>> job = submitit.SlurmJob("log_test", "33389760")
>>> job.submission()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/home/louismartin/dev/ext/submitit/submitit/core/core.py", line 208, in submission
    return utils.DelayedSubmission.load(self.paths.submitted_pickle)
  File "/private/home/louismartin/dev/ext/submitit/submitit/core/utils.py", line 135, in load
    obj = pickle_load(filepath)
  File "/private/home/louismartin/dev/ext/submitit/submitit/core/utils.py", line 273, in pickle_load
    return pickle.load(ifile)
TypeError: an integer is required (got type bytes)
>>> job.submission()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/home/louismartin/dev/ext/submitit/submitit/core/core.py", line 208, in submission
    return utils.DelayedSubmission.load(self.paths.submitted_pickle)
  File "/private/home/louismartin/dev/ext/submitit/submitit/core/utils.py", line 135, in load
    obj = pickle_load(filepath)
  File "/private/home/louismartin/dev/ext/submitit/submitit/core/utils.py", line 273, in pickle_load
    return pickle.load(ifile)
TypeError: an integer is required (got type bytes)

[louismartin]

Actually this seems to be a bug with cloudpickle, simpler repro:

In python 3.7 (cloudpickle==1.6.0):

import cloudpickle

def foo():
    pass

with open("foo.pkl", "wb") as ofile:
    cloudpickle.dump(foo, ofile)

In python 3.8 (cloudpickle==1.6.0):

import cloudpickle

with open("foo.pkl", "rb") as ifile:
    print(cloudpickle.load(ifile))

Throws

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
TypeError: an integer is required (got type bytes)

I’ll create an issue in cloudpickle.

[gwenzek]

Works for me. I’ll need more details to help you.

import pickle
import sys
from pathlib import Path

import submitit
from submitit.auto.auto import AutoExecutor

PKL_36 = Path("./python36.pkl")
PKL_38 = Path("./python38.pkl")


def dump():
    pkl = PKL_36 if sys.version_info.minor == 6 else PKL_38

    ex = submitit.AutoExecutor("logs")
    pkl.write_bytes(pickle.dumps(ex, protocol=4))


def load():
    pkl = PKL_38 if sys.version_info.minor == 6 else PKL_36
    ex = pickle.loads(pkl.read_bytes())

    assert isinstance(ex, AutoExecutor), f"Wasn't able to load pickle from {pkl} to {sys.version_info}"
    print(f"Loaded pickle from {pkl} to python {sys.version_info.major}.{sys.version_info.minor}")


if __name__ == "__main__":
    dump()
    load()

I ran the above script with two versions of python, 3.8 and 3.6. Both environment had submitit installed. Submitit isn’t required but I wanted to pickle a non trivial object instead of just a string.

➜ ./venv36/bin/python pickle_version.py 
FileNotFoundError: [Errno 2] No such file or directory: 'python38.pkl'
➜ python pickle_version.py                                          
Loaded pickle from python36.pkl to python 3.8                                   
(base)                                                                          
➜ ./venv36/bin/python pickle_version.py                                          
Loaded pickle from python38.pkl to python 3.6