Problem Finetuning with Provided Pretrained Model

Hi, I am encountering the following errors recently when I try to finetune using the provided pretrained models.

1. When I cloned the original repo and tried finetuning on CORD per the instructions like below: python train.py --config config/train_cord.yaml \ --pretrained_model_name_or_path "naver-clova-ix/donut-base" \ --dataset_name_or_paths '["naver-clova-ix/cord-v2"]' \ --exp_version "test_experiment" the following error pops up: Traceback (most recent call last): File "train.py", line 149, in <module> train(config) File "train.py", line 130, in train callbacks=[lr_callback, checkpoint_callback], File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/argparse.py", line 345, in insert_env_defaults return fn(self, **kwargs) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py", line 459, in __init__ training_epoch_loop = TrainingEpochLoop(min_steps=min_steps, max_steps=max_steps) File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 51, in __init__ if max_steps < -1: TypeError: '<' not supported between instances of 'NoneType' and 'int'

2. When I tried with my local modified repo, the following error pops up Traceback (most recent call last): File "train.py", line 146, in <module> train(config) File "train.py", line 57, in train model_module = DonutModelPLModule(config) File "/data/project/users/xingjianzhao/visual-information-extraction/code/Donut/donut/donut/lightning_module.py", line 94, in __init__ self.model = DonutModel.from_pretrained( File "/data/project/users/xingjianzhao/visual-information-extraction/code/Donut/donut/donut/donut/model.py", line 642, in from_pretrained model = super(DonutModel, cls).from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py", line 2155, in from_pretrained model, missing_keys, unexpected_keys, mismatched_keys, error_msgs = cls._load_pretrained_model( File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py", line 2282, in _load_pretrained_model model._init_weights(module) File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_utils.py", line 1050, in _init_weights raise NotImplementedError(f"Make sure_init_weightsis implemented for {self.__class__}") NotImplementedError: Make sure_init_weightsis implemented for <class 'donut.model.DonutModel'> While I did made some modifications, I tried with previous versions of my repo that worked perfectly fine and this error still pops up. However when I use my previous finetuned models (trained with the exact same code), it works fine. I’m wondering if you may have an idea on what the problems could be. Thanks!

[leitouran]

Oh! The “official” branch was literally a branch named “official” haha. I’ll try that out first thing tomorrow. Thank you @gwkrsrch!

[gwkrsrch]

Hi @leitouran @xingjianz , thanks for reporting the issues with rich contexts/information.

The first issue is resolved with 95cde5a.

For the second issue in (2), It seems wrong weights are fed to the donut-python. The error seems to pop up when model weights and an initialized donut-python model architecture did not match. To be specific, for @leitouran , I recommend seeing these release notes at https://github.com/clovaai/donut/releases/tag/1.0.5. Cloning from the official branch would fix the issue.

Hope this helps. Please let me know if you are still confused. Feel free to reopen this or open another issue if you have anything new for sharing and debugging 😃