Trying to pretrain with longer max sequence length

I’ve tried to extend your electra openwebtext examples by using your pre-process script with appropriate arguments for max sequence length of 768, as follows:

export PYTHONPATH=.
python pretraining/openwebtext/preprocess.py \
--max-seq-length 768 \
--trg-dir data/openwebtext_features_768 \
--n-dataset-building-processes 8

I then have custom json files for the electra generator and discriminator with the following change from the small_generator.json and small_discriminator.json:

"embedding_size": 768,

When I try to do pretraining with the new tokenization and the changed specification I get what looks like a dimension mismatch during training. I have been careful to pass the modified data folder with the new, longer feature tensors.

[lucidrains]

@DarrenAbramson I think I may have spotted the error. So the "max_position_embeddings": 512 needs to be increased to 768 as well, or you will hit an out of bounds when trying to get that positional encoding greater than the default (512)

[lucidrains]

No problem, I’ll see if I can add a line of code to guard against that later tonight. Thanks for trying this out!