Long segmentation time on large microscopy images

Disclaimer This issue is based on preliminary observations and does not contain a lot of details/examples at the moment.

Issue description

I noticed that the segmentation time on large microscopy images is quite long in ivadomed compared to what I was used to in ADS. To this effect, I fixed a bug in image reconstruction in PR #956. In addition, we found that using the .pt version of the model (on GPU) and the .onnx version (on CPU) help with segmentation time as described in issue #961.

Nevertheless, for a microscopy image of ~8000x6000 pixels, the segmentation time with my ivadomed model (.pt on GPU on rosenberg) is around 2.5 minutes. In comparison, an ADS model (on GPU on bireli) would take less than 30 seconds to segment the same image.

I investigated a little bit and the bottleneck seems to come from this for loop. More specifically, there is a delay between the return of reconstruct_3d_object and the beginning of the next iteration of the loop (is it the time to fetch the batch in DataLoader?)

I could use some help here, thanks!

[kousu]

Update: I fixed an issue in PR #982 where the transforms were applied on the entire image for each patch instead of on each individual patch. This reduce the segmentation time from 171 sec to 118 sec.

Wow! Congratulations that’s already an amazing performance boost. 🎉 🎉

[kousu]

And @dyt811 I don’t think you need to spend your time profiling with work-in-progress branch afterall.