Can we add an augment `min_new_tokens` to the `generate` function?

Feature request

Can we add a new parameter min_new_tokens to the generate function to limit the length of newly generated tokens? The current parameter min_length limits the length of prompt + newly generated tokens, not the length of newly generated tokens.

Motivation

We already have max_new_tokens to limit the max length of the generated tokens, i.e., max_length = max_new_tokens + prompt.

Why not add the min_new_token to limit the min length of the generated tokens? (i.e., min_length = min_new_tokens + prompt)

I know this is kind like an other syntax sugar, but it will be much convenient if we have this parameter.

Your contribution

I can sumbit a PR.

[silverriver]

@PanQiWei I agree, but I think it is impossible to change the current implementation of max_length and min_length for the conern of back compatibility.

[PanQiWei]

For my understanding, in the current implementation, min_length set the length limit of len(promt) + len(generated tokens)

See the implementation of MinLengthLogitsProcessor:

    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
        cur_len = input_ids.shape[-1]
        if cur_len < self.min_length:
            scores[:, self.eos_token_id] = -float("inf")
        return scores

input_ids in the previous code block refers to prompt + generated tokens. For example, see the implemented of some decoding method for how logits processors are called. (See beam_search() or greedy_decoding())

It will be more convenient if we set an argument min_new_tokens to only limit the length of generated tokens, not prompt + generated tokens.

Please correct me if I have missed something

@silverriver yes you are right, its my mistake 😂

But I still think min_new_tokens and min_length should mean the same thing and also to max_new_tokens and max_length (though they are actually different now), because most people who use model.generate would think min_length means to ‘at least generate min_length tokens’ and max_length means to ‘generate tokens no more than max_length’