What's kernel launch time?

The performance summary shows that my model spend ~50% time in the “kernel launch” step. I find other items easy to understand, but I have no idea what “kernel launch” is, and how I can reduce its time consumption. I do complicated preprocessing to my data using the tf.data.Dataset APIs, but the summary shows that I spend no time on it. Could the preprocessing be the real cause of the high “kernel launch” time consumption?

Thanks!

[ckluk]

Hi Slyne,

Usually, high kernel launch overhead is not due to a particular op. Instead, it is because there is insufficient computation in the ops executed on the device to amortize the kernel launch overhead. So, following suggestions from one of my previous replies may still be helpful: (1) Use the XLA compiler, which can fuse multiple TF Ops into the same kernel (see https://www.tensorflow.org/xla). (2) Rewrite your TF code in a way to create larger granularity of computation on the GPU. In particular, have you tried (1)?

On Wed, Jan 13, 2021 at 4:35 AM Slyne Deng [e-mail] wrote:

@ckluk https://github.com/ckluk Hi, I also encounter this problem. After checking the profiles uploaded, it’s still difficult for me to find the exact operations that I should edit in my codes. It seems there are frequent ops between host and device. Not sure if my guess is correct. Could you help me? Uploading 2021_01_13_19_07_07.zip… [image: StepTime] https://user-images.githubusercontent.com/6286804/104453007-be601280-55de-11eb-8498-8a4d4649cd85.JPG

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tensorflow/profiler/issues/8#issuecomment-759420428, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE33L3OORZMZ4SVVSKQZ6H3SZWHR7ANCNFSM4MMH2VAA .

[ckluk]

Thanks for providing the log. After inspecting it, I believe that the kernel-launch time is actually overlapped with the tf.data preprocessing time on the CPU. It is possible that the tf.data preprocessing interferes with the kernel launching. You can’t do much with the kernel launching, but you can try to reduce the interference of tf.data preprocessing by: (1) Using more parallelism in tf.data preprocessing (https://www.tensorflow.org/guide/data_performance) (2) Using a more powerful CPU (3) Offloading some of the data preprocessing to GPU