Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

no attribute 'tokenize_one_example'

See original GitHub issue

Hi,

thank you for your amazing work to ease the users of prompt learning.

I tried to implement the LMBFF tutorial and happened to meet this error:

Traceback (most recent call last):
  File "run_lmbff.py", line 116, in <module>
    dataloader = PromptDataLoader(dataset['train'], template, template_generate_tokenizer, template_tokenizer_wrapper, batch_size=len(dataset['train']), decoder_max_length=128) # register all data at once
  File "/home/oryza/playground/OpenPrompt/openprompt/pipeline_base.py", line 101, in __init__
    self.tokenize()
  File "/home/oryza/playground/OpenPrompt/openprompt/pipeline_base.py", line 137, in tokenize
    inputfeatures = InputFeatures(**self.tokenizer_wrapper.tokenize_one_example(wrapped_example, self.teacher_forcing), **wrapped_example[1]).to_tensor()
AttributeError: 'T5Tokenizer' object has no attribute 'tokenize_one_example'

This is my pip list:

Package            Version   Editable project location
------------------ --------- ---------------------------------
aiohttp            3.8.1
aiosignal          1.2.0
async-timeout      4.0.2
asynctest          0.13.0
attrs              21.4.0
certifi            2021.10.8
charset-normalizer 2.0.12
click              8.1.2
datasets           2.0.0
dill               0.3.4
filelock           3.6.0
frozenlist         1.3.0
fsspec             2022.3.0
huggingface-hub    0.5.1
idna               3.3
importlib-metadata 4.11.3
joblib             1.1.0
multidict          6.0.2
multiprocess       0.70.12.2
nltk               3.7
numpy              1.21.5
openprompt         1.0.0     /home/oryza/playground/OpenPrompt
packaging          21.3
pandas             1.3.5
pip                22.0.4
protobuf           3.20.0
pyarrow            7.0.0
pyparsing          3.0.8
python-dateutil    2.8.2
pytz               2022.1
PyYAML             6.0
regex              2022.3.15
requests           2.27.1
responses          0.18.0
rouge              1.0.0
sacremoses         0.0.49
scikit-learn       1.0.2
scipy              1.7.3
sentencepiece      0.1.96
setuptools         41.2.0
six                1.16.0
sklearn            0.0
tensorboardX       2.5
threadpoolctl      3.1.0
tokenizers         0.10.3
torch              1.11.0
tqdm               4.64.0
transformers       4.10.0
typing_extensions  4.1.1
urllib3            1.26.9
xxhash             3.0.0
yacs               0.1.8
yarl               1.7.2
zipp               3.8.0

Do you have any idea about the error? I read in another thread about installing SentencePiece to solve this problem but my sentencepiece is already there.

Thank you in advance!

Best, Oryza

Issue Analytics

State:
Created a year ago
Comments:6 (2 by maintainers)

Top GitHub Comments

2reactions

cyl628commented, Apr 20, 2022

Hi,

thank you for your amazing work to ease the users of prompt learning.

I tried to implement the LMBFF tutorial and happened to meet this error:

Traceback (most recent call last):
  File "run_lmbff.py", line 116, in <module>
    dataloader = PromptDataLoader(dataset['train'], template, template_generate_tokenizer, template_tokenizer_wrapper, batch_size=len(dataset['train']), decoder_max_length=128) # register all data at once
  File "/home/oryza/playground/OpenPrompt/openprompt/pipeline_base.py", line 101, in __init__
    self.tokenize()
  File "/home/oryza/playground/OpenPrompt/openprompt/pipeline_base.py", line 137, in tokenize
    inputfeatures = InputFeatures(**self.tokenizer_wrapper.tokenize_one_example(wrapped_example, self.teacher_forcing), **wrapped_example[1]).to_tensor()
AttributeError: 'T5Tokenizer' object has no attribute 'tokenize_one_example'

This is my pip list:

Package            Version   Editable project location
------------------ --------- ---------------------------------
aiohttp            3.8.1
aiosignal          1.2.0
async-timeout      4.0.2
asynctest          0.13.0
attrs              21.4.0
certifi            2021.10.8
charset-normalizer 2.0.12
click              8.1.2
datasets           2.0.0
dill               0.3.4
filelock           3.6.0
frozenlist         1.3.0
fsspec             2022.3.0
huggingface-hub    0.5.1
idna               3.3
importlib-metadata 4.11.3
joblib             1.1.0
multidict          6.0.2
multiprocess       0.70.12.2
nltk               3.7
numpy              1.21.5
openprompt         1.0.0     /home/oryza/playground/OpenPrompt
packaging          21.3
pandas             1.3.5
pip                22.0.4
protobuf           3.20.0
pyarrow            7.0.0
pyparsing          3.0.8
python-dateutil    2.8.2
pytz               2022.1
PyYAML             6.0
regex              2022.3.15
requests           2.27.1
responses          0.18.0
rouge              1.0.0
sacremoses         0.0.49
scikit-learn       1.0.2
scipy              1.7.3
sentencepiece      0.1.96
setuptools         41.2.0
six                1.16.0
sklearn            0.0
tensorboardX       2.5
threadpoolctl      3.1.0
tokenizers         0.10.3
torch              1.11.0
tqdm               4.64.0
transformers       4.10.0
typing_extensions  4.1.1
urllib3            1.26.9
xxhash             3.0.0
yacs               0.1.8
yarl               1.7.2
zipp               3.8.0

Do you have any idea about the error? I read in another thread about installing SentencePiece to solve this problem but my sentencepiece is already there.

Thank you in advance!

Best, Oryza

This is probably due to misplaced positional parameters. Already fix it.

0reactions

khairunnisaorcommented, Apr 22, 2022

Thanks for the update!

About the wrapper problem, I think I will open a new issue to ask about how the wrapper works. Thank you so much for your fast and helpful responses!