Error while saving Longformer pre-trained model

Thanks for the transformers library!

Information

I am trying to finetune a pre-trained model of type LongformerForQuestionAnswer on a custom QA dataset using a custom script morphed from run_squad.py. The pre-trained model is allenai/longformer-large-4096-finetuned-triviaqa

While saving the pretrained model, I run into the following error:

Traceback (most recent call last):
  File "examples/question-answering/run_nq.py", line 809, in <module>
    main()
  File "examples/question-answering/run_nq.py", line 752, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "examples/question-answering/run_nq.py", line 248, in train
    tokenizer.save_pretrained(output_dir)
  File "/home/danishp/git/explain-qa/src/third_party/transformers/src/transformers/tokenization_utils_base.py", line 1368, in save_pretrained
    write_dict[key] = value.__getstate__()
AttributeError: 'AddedToken' object has no attribute '__getstate__'

[gabrer]

Actually, I never pip-installed the tranformers library, I am just running the cloned github code from a few days ago (this is because I had to edit some parts of the code for my use case).

However, when I pip installed these versions, surprisingly, I don’t see this error. As you suggest, it is possible that some tokenizer changes that were intended for version 3.0.0 crept in.

In the cloned code that I am using, if I change the following line to:

https://github.com/huggingface/transformers/blob/ef0e9d806c51059b07b98cb0279a20d3ba3cbc1d/src/transformers/tokenization_utils_base.py#L1368

write_dict[key] = value.content # instead of __getstate__()

The problem is fixed.

Just had the same issue with version 3.0.2 while fine-tuning the Robert-base model. Guess, it would have been the same with other BERT-base models. Changing this line solved the issue.

[danishpruthi]

Actually, I never pip-installed the tranformers library, I am just running the cloned github code from a few days ago (this is because I had to edit some parts of the code for my use case).

However, when I pip installed these versions, surprisingly, I don’t see this error. As you suggest, it is possible that some tokenizer changes that were intended for version 3.0.0 crept in.

In the cloned code that I am using, if I change the following line to:

https://github.com/huggingface/transformers/blob/ef0e9d806c51059b07b98cb0279a20d3ba3cbc1d/src/transformers/tokenization_utils_base.py#L1368

write_dict[key] = value.content # instead of __getstate__()

The problem is fixed.