Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Add the fast implementation of `BlenderbotTokenizer`

See original GitHub issue

🚀 Feature request

As it is the case for other models’ tokenizers, add the fast implementation of BlenderbotTokenizer.

Motivation

To have faster tokenization for Blenderbot models. (Also, the implementation should be pretty straightforward considering the similarity to the RobertaTokenizer/RobertaTokenizerFast.)

Your contribution

I would like to have a look at this and will be glad to add that.

Issue Analytics

State:
Created 2 years ago
Comments:6 (5 by maintainers)

Top GitHub Comments

1reaction

LysandreJikcommented, Sep 21, 2021

Should be done with huggingface#c468b23

1reaction

LysandreJikcommented, Sep 21, 2021

Ah ! There’s no way to do that as of now - let me handle that for you.

Top Results From Across the Web

Blenderbot - Hugging Face

Construct a “fast” Blenderbot tokenizer (backed by HuggingFace's tokenizers library), derived from the GPT-2 tokenizer, using byte-level Byte-Pair-Encoding.

transformers/modeling_blenderbot.py at main · huggingface ...

add present self-attn cache to positions 1,2 of present_key_value tuple ... Example: ```python. >>> from transformers import BlenderbotTokenizer, ...

Creating A Telegram Chatbot Using BlenderBot From Meta AI

A next step is to access BlenderBot via a HuggingFace inference API in a Notebook environment. And lastly I look at creating a...

How to Build a Conversational AI bot Using Blenderbot

To begin with, we are going to import the model class and the tokenizer. from transformers import BlenderbotTokenizer, ...

Build a Conversation Agent using Facebook's Blenderbot and ...

Ever wanted to talk to a real life AI model? Well using Blenderbot you can...and it's NOT even that hard. In this tutorial...