Empty Conversation Responses

Environment info

Mac OS 10.14

My work is in rust, and I have an issue open in guillaume-be/rust-bert#87 however the authour of the repo asked that I also open it here to get HugginFace’s opinion as it pertains to code that matches the intention of that in this repo.

• transformers version: rust-bert 0.10.0
• Platform: Mac OS
• PyTorch version (GPU?): No
• Tensorflow version (GPU?): No
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: Yes

Who can help

TextGeneration: @TevenLeScao (This is in how min_ and max_length are treated during text generation of the conversation model)

Information

Model I am using: DialgoGPT with the Conversation Moel

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The problem occurs in the offical example of the rust and the owner of the the rust code assures me that there is exact same behaviour in this code too

The tasks I am working on is:

• my own task or dataset: (give details below)

I am making a small chatbot, I have my own fine tuned model but the same behaviour is obseved with the stock DiagloGPT model

To reproduce

Steps to reproduce the behavior:

1. Create a conversation model with min_length set
2. Talk with the conversation for about 10-12 responses
3. Responses will be zero length despite min_length begin set

Expected behavior

Min length to be upheld during conversation

Details

The root cause of this is how cur_len and min/max_len are handled in the code

https://github.com/huggingface/transformers/blob/15a189049e01fbd3bef902848a09f58a5f006c37/src/transformers/generation_utils.py#L86-L88

https://github.com/huggingface/transformers/blob/15a189049e01fbd3bef902848a09f58a5f006c37/src/transformers/generation_utils.py#L533

The cur_len is initalised with the length of current input which contains all previous dialogue with the bot as context

https://github.com/huggingface/transformers/blob/15a189049e01fbd3bef902848a09f58a5f006c37/src/transformers/generation_utils.py#L451

This means that min_length of the new utterence from the bot is already satisfied. It also means that max_length can be exceeded if a long conversation is held.

cur_len should perhaps be initialised differently in the ConverstaionModel

[sshleifer]

My opinion is that I have never seen “users might want to generate multiple articles < 100 words including the prefix”, and I have seen, “Why does prefix count?” a number of times and had the same misunderstanding the first few times I used it. So I think we should change the behavior to not count decoder_start_token_id or tokens in input_ids. Another argument that might land: you would expect a function called generate to generate N tokens. Not generate N tokens where N varies based on your inputs.

The use_relative compromise is my second favorite option. The status quo is my least favorite.

[QuantumEntangledAndy]

I do not believe this will resolve the issue with min_length. Surly the expected behavior of min_length in a conversation model would be the min length of the new utterence not the length of context + new utterence.

Lets say I set min length to 2, because I want my bot to always say something and I say, “What shall we watch?” This has a length of 4 (five with with eos) this is added into the context and counts towards the length for min_length. The bot therefore has every chance of saying nothing.

What would be the ideal way to deal with this situation?