Benchmarking methodology used is not quite correct

Hi @1ytic ,

Thank you for this work on optimizing warp rnn-t operation which is becoming increasingly useful for many speech recognition acoustic models. We have studied your implementation and have the following observations:

1. When we run the benchmark script in your repo, the run time we got is as below, where new refers to this new repo, and the baseline is what we have now in RNN-T reference model. We used B=32, T=U=200, V= 29, which is a typical case in our dataset. From the output of the benchmark script, it does appear that the new loss function runs faster:

new: 1.76 ms baseline: 6.10 ms

1. However, in the benchmark script that the author provided, the run time was measured as:

t = timer()
costs = loss(xs, ys, xn, yn)
elapsed_time += timer() - t

This way of measuring has a problem that CPU could run ahead of GPU and stop the timer even before the kernel is completed. After adding synchronization as below,

torch.cuda.synchronize() # sync before start the timer
t = timer()
costs = loss(xs, ys, xn, yn)
torch.cuda.synchronize() # sync before stop the timer
elapsed_time += timer() - t

the run time we get is:

new: 15.82 ms baseline: 6.12 ms

1. This is similar to the run time we got from GPU profiler nsys:

new: 14.38 ms baseline: 4.75 ms

In summary - It does not look like that the alternative loss function is running faster than what we have. The claimed speedup in the repo is likely caused by a flawed benchmark methodology.

Can you share your thought process on this?

Thanks, Ashish

[dophist]

It was tested on 2080ti

[1ytic]

I added one more explanation 2944982 of performance issue with NVIDIA Profiler.