DeepSpeed and Generate Method

Hi @lucidrains

I’m currently testing the generate function of the TrainingWrapper class. When I use DeepSpeed and I try to generate a sequence it gives me the following error: AttributeError: 'DeepSpeedLight' object has no attribute 'generate'

Is it because Generation can only be done outside DeepSpeed Engine?

Thank you very much, once again! 😃

[CalogeroZarbo]

I understand @lucidrains. The frustration is at max when something, where you put so much effort, does not work as expected.

However, I’m going to keep doing experiments and test the library behavior under different circumstances and see how it goes. It may be the case that for longer sequences this is the only way we can address the memory issue. For instance, I do understand that Reformer, as you say, do not learn as well as full attention, but in many cases, for longer sequences, you cannot even think to use a full attention transformer. At the end of the day, it’s better to have a decrease in performance rather than being unable to perform the experiments and develop the solution.

With all that said, I still believe there is much value in your work, and in the time you generously spent in this project. Me as well as all the community we are very grateful to you.

Thank you again for your tips & tricks!

Cheers, Cal

[CalogeroZarbo]

Hi @lucidrains ! Thank you for your great news! No need to apologize, this is the hearth of open source: all the users looking your code help to improve it and discover mistakes here and there, at the end of the day we are all human being.

I also plan to use the Sinkhorn repo you made, and I’m very curious to see it’s performance. If you would have to choose, for long sequences which technique would you recommend? Reformer or your implementation of the Sinkhorn attention? I saw that in the latter you also add reversibility and chunking techniques. Which repo would you use for Enc/Dec architecture?

Thank you so much for your effort!