Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Encoder-decoder model is not working correctly for the latest versions

See original GitHub issue

System Info

transformers==4.2.1

Who can help?

@patrickvonplaten

Information

The official example scripts
My own modified scripts

Tasks

[] An officially supported task in the examples folder (such as GLUE/SQuAD, …)
[X ] My own task or dataset (give details below)

Reproduction

Hi,

I’m working with a seq2seq problem, in particular with EncoderDecoderModel model. The problem is that I can’t have good results with the latest version (4.21.3). I also tried with 4.18.0 because of this but didn’t work either. It is however working when using version 4.2.1

I have made an public notebook you can run to see the issue. Is an example to train a model to generate the written digits, given a number.

Expected behavior

You works nicely with version 4.2.1, but very bad with the most recent versions.

Issue Analytics

State:
Created a year ago
Comments:8 (4 by maintainers)

Top GitHub Comments

2reactions

ArthurZuckercommented, Dec 6, 2022

Hey! Sorry not yet, it’s pretty tricky, but I hope I’ll resolve it soon! 🤗

1reaction

ArthurZuckercommented, Sep 21, 2022

Hey @miguelwon it seems that you are right about the training not converging at all using current version. However, since loading a trained model in the new versions does not give bad results, I suspect that the issue comes from either the computation of the loss, or the trainer.

I will have a look in more details as I believe this is a pretty important bug 😄

Top Results From Across the Web

Encoder Decoder Models - Hugging Face

The EncoderDecoderModel can be used to initialize a sequence-to-sequence model with any pretrained autoencoding model as the encoder and any pretrained ...

LSTM encoder decoder model training errors: ValueError

My LSTM Encoder Decoder model takes 2 input as sequence: items and item-types ... Last dense layer calculates probability of purchase of 129 ......

How to Develop an Encoder-Decoder Model for Sequence-to ...

How to correctly define a sophisticated encoder-decoder model in Keras ... I had the same problem, and updated Keras (to version 2.1.2) and ......

Encoder-Decoder Seq2Seq Models, Clearly Explained!!

The Encoder-Decoder architecture is relatively new and had been ... problems for which modified versions of this RNN architecture are used.

10.7. Encoder-Decoder Seq2Seq for Machine Translation

In so-called seq2seq problems like machine translation (as discussed in Section ... one token at a time, the decoder model, consisting of a...