Different behaviour when extending this project to Bart

Hello there. This is a really fantastic project. I’m trying to extend your work to Bart but I’ve run into some strange behaviour.

I’ve made a Colab notebook to illustrate the problem. Specifically when converting Bart to ONNX, the encoder_hidden_states input does not get included in the ONNX model’s graph. As you can see from the notebook though, it works perfectly for T5.

I realise this is out of scope for the fastT5 project but thought someone who comes across this issue might have experienced a similar problem and be able to help. This may also be useful to know in case you have plans to expand this project to include models like Bart in the future.

[tobigue]

Hey, I also came across this when trying to adapt the fastT5 code for converting (m)BART to onnx and I think it is due to the fact that the encoder_hidden_states are passed as key_value_states into the MBartAttention down the line and are not used in case past_key_value is given.

So my guess was, that it is not included in the exported graph for that reason. I’ll try to share my progress in a Colab notebook tomorrow.

[HenryDashwood]

Very odd error that I also get sometimes and my only answer at this point is… have you tried running the cell again? That usually sorts it