Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

data_collator.py - line 326, in mask tokens - xlnet finetuning error

See original GitHub issue

Environment info

transformers version: version: 3.2.0
Platform: Linux-4.15.0-118-generic-x86_64-with-Ubuntu-18.04-bionic
Python version: 3.6.9
PyTorch version (GPU?): 1.5.1+cu101 (True)
Tensorflow version (GPU?): not installed (NA)
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No

Who can help

TransfoXL/XLNet: @TevenLeScao

Information

Model I am using (XLNet):

The problem arises when using:

[ x] the official example scripts: (give details below)
my own modified scripts: (give details below)

Testing simple example in ‘language-modeling/examples/README’ using recommended wiki-2-raw dataset and xlnet-base cased model

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name)
[x ] my own task or dataset: (give details below)

Same error occurs using simple one sentence per line text file (10 megs)

To reproduce

Steps to reproduce the behavior:

Run all steps in ‘language-modeling/examples/README’ using xlnet-base-cased (cached or local)
Model loads with warnings and process begins before quickly exiting with the following error:

File “/home/pixelhead/Desktop/xlnet/transformers-master/transformers/data/data_collator.py”, line 326, in mask_tokens “This collator requires that sequence lengths be even to create a leakage-free perm_mask. Please see relevant comments in source code for details.” ValueError: This collator requires that sequence lengths be even to create a leakage-free perm_mask. Please see relevant comments in source code for details. Epoch: 0%| | 0/3 [00:00<?, ?it/s] Iteration: 0%|

Expected behavior

Expect ‘run_language_modeling.py’ to work for xlnet as per ‘language-modeling/examples/README’ Have tested addition of ‘–line_by_line’ and ‘block_size=128, 256, 512’ etc. Same error.

Could be missing something here ‘Please see relevant comments in source code for details.’ but not clear.

Cheers,

Issue Analytics

State:
Created 3 years ago
Comments:9 (1 by maintainers)

Top GitHub Comments

3reactions

mihaidobricommented, Sep 28, 2020

I have the same issue. Does anybody know any “workarounds” to bypass this issue?

0reactions

darwinhariantocommented, Jul 13, 2021

I ended up adding <pad> if token length is not even. Is this ok?

Top Results From Across the Web

XLNet - Hugging Face

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the...

pip install transformers==2.0.0 - PyPI

run_squad.py : an example fine-tuning Bert, XLNet and XLM on the question answering dataset SQuAD 2.0 (token-level classification); run_generation.py : an ...

XLNet Fine-Tuning Tutorial with PyTorch - Chris McCormick

In this tutorial, I'll show you how to finetune the pretrained XLNet model with ... Whereas with XLNet the token pattern looks like...

Fine-Tuning XLNet for Sentiment Analysis - Nana Boateng

XLNet employs Transformer-XL autoregressive model into pre-training ... prediction will be the last token of the sequence (the masked token).

How to use XLNET from the Hugging Face transformer library

Secondly, when BERT masks more than one token in a sentence, ... from the Hugging Face transformer library without any fine-tuning/training.