Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

torch.save fails with pickler.dump for bytes

See original GitHub issue

❓ Questions and Help

This is probably not an issue with pytorch. I try to torch.save an ops with bytes from pybind11 pickle. Here is the pybind11 registration link.

How do I torch.save the ops:

import torchtext
import torch
from torchtext.experimental.transforms import  PRETRAINED_SP_MODEL, load_sp_model
sp_model_path = torchtext.utils.download_from_url(PRETRAINED_SP_MODEL['text_unigram_25000'])
sp_model = load_sp_model(sp_model_path)
f = open("temp.pt", "wb")
torch.save(sp_model, f)

The error message I got:

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-7-c66cdf260b7d> in <module>
      1 f = open("temp.pt", "wb")
----> 2 torch.save(sp_model, f)

~/tmp/PyTorch/pytorch/torch/serialization.py in save(obj, f, pickle_module, pickle_protocol, _use_new_zipfile_serialization)
    370         if _use_new_zipfile_serialization:
    371             with _open_zipfile_writer(opened_file) as opened_zipfile:
--> 372                 _save(obj, opened_zipfile, pickle_module, pickle_protocol)
    373                 return
    374         _legacy_save(obj, opened_file, pickle_module, pickle_protocol)

~/tmp/PyTorch/pytorch/torch/serialization.py in _save(obj, zip_file, pickle_module, pickle_protocol)
    474     pickler = pickle_module.Pickler(data_buf, protocol=pickle_protocol)
    475     pickler.persistent_id = persistent_id
--> 476     pickler.dump(obj)
    477     data_value = data_buf.getvalue()
    478     zip_file.write_record('data.pkl', data_value, len(data_value))

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 119: invalid start byte

cc @mruberry