How to prevent tokenizer from outputting certain information

Be aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'longest_first' truncation strategy. So the returned list will always be empty even if some tokens have been removed.

[eduOS]

Set the verbosity level as follows:

transformers.logging.set_verbosity_error()

[lextoumbourou]

This warning can be pretty noisy when your batch size is low, and the dataset is big. It would be nice only to see this warning once, as nreimers mentioned.

---

For anyone coming from Google who cannot suppress the error with eduOS’s solution. The nuclear option is to disable all warnings in Python like this:

import logging
logging.disable(logging.WARNING)