Filling more than 1 masked token at a time

I am able to use hugging face’s mask filling pipeline to predict 1 masked token in a sentence using the below:

!pip install -q transformers
from __future__ import print_function
import ipywidgets as widgets
from transformers import pipeline

nlp_fill = pipeline('fill-mask')
nlp_fill("I am going to guess <mask> in this sentence")

But does anyone have an opinion on what is the best way to do this if I want to predict 2 masked tokens? e.g. if the sentence is instead "I am going to <mask> <mask> in this sentence"?

If i try and put this exact sentence into nlp_fill I get the error “ValueError: only one element tensors can be converted to Python scalars” so it doesn’t work automatically.

Any help would be much appreciated!

Stack overflow question link

[mitramir55]

Hi, I’ve implemented right to left, left to right, and random mask filling in PyTorch for top k ids that the model thinks are the most probable tokens in a sentence in one of my projects. In this implementation, each time we want to generate a mask, the model looks at the previously generated sentences and decides what is the most probable for the next masked position. So if we have 2 masks in a sentence, by setting top_k=5, we’ll have 25 sentences (5 tokens for the first position, and for each of these 5 sentences with one mask we have another 5 tokens for the second mask). It’ll output something like this:(I used Persian models for this. I hope you can see how the masks are being filled) Then in the next step, we implemented a beam search to choose the most probable sequence of all between all these sentences.

I’d be glad to help HuggingFace on this issue, I can send my code or send a pull request.

[LysandreJik]

Please see the following issue https://github.com/huggingface/transformers/issues/10158 and PR https://github.com/huggingface/transformers/pull/10222 for an attempt to take a crack at this