BART.generate: possible to reduce time/memory?

🐛 Performance issues

I did a quick benchmark between HuggingFace’s implementation of BART and FairSeq’s implementation.

You can find the benchmark code here.

---

Here is my results, on a single GPU GTX 1080 (12 GiB of memory) :

FP16 - Batch size 16	s/batch	s/sample
FairSeq	8.8676	0.5664
HuggingFace	12.3358	0.7879

FP16 - Batch size 32	s/batch	s/sample
FairSeq	17.1247	0.5469
HuggingFace	OOM	OOM

FP16 - Batch size 1	s/sample
FairSeq	1.6743
HuggingFace	1.8856

FP32 - Batch size 1	s/sample
FairSeq	1.7865
HuggingFace	2.0670

---

FairSeq is consistently faster than HuggingFace on all my experiments.

---

This sparks a few questions :

• Do you have similar results on your side ? Did I mess my benchmark ?
• Why HuggingFace’s implementation is significantly slower ?
• Why HuggingFace’s implementation takes more space in memory (illustrated by OOM with batch size of 32) ?
• Is the release of the Summarization Pipeline going to improve this ?

@sshleifer

[sshleifer]

On master, the gap has closed considerably! <16GB GPU RAM for fp16, bs=32, and timings much closer:

My numbers are a bit lower than yours because I am on an NVIDIA RTX GPU.

[sshleifer]

1. Identical to my benchmark for speed. Hadn’t tested memory but I’m not surprised that their implementation is less.

For both memory and speed, they have a lot of clever tricks that we haven’t implemented yet.

1. Summarization Pipeline will not help, but I will take a longer look at this tomorrow and see if we can improve.