Unable to pass encoder_outputs to generate calls

Environment info

• transformers version: Github Main branch

Who can help

TextGeneration: @TevenLeScao

Information

Model I am using (Bert, XLNet …): T5

I’m unable to pass preconputed encoder_outputs to the .generate() method.

I’ve tried defining:

model_kwargs = {"encoder_outputs": encoder_outputs}
output = model.generate(model_kwargs)

But I’ve noticed some validation errors for input_ids. Even if I replace input_ids with dummy values, I’ve noticed the model_kwargs is always replaced here:

https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L448

I think it can be fixed (without optimization) by just replacing:

model_kwargs["encoder_outputs"] = encoder_outputs

with:

model_kwargs.setdefault("encoder_outputs", encoder_outputs)

If you agree I can try to open a PR to fix this.

Best,

[patrickvonplaten]

Hey @gabisurita - I understand that one might want to forward encoder_outputs in the generate function. However, adding such a possibility opens the door for many problems in case beam_search is chosen. We are currently working on a bigger refactor that should solve this problem by a better design choice of generate(). I’m afraid that this will still take ~3,4 weeks to complete though.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.