Performance of extract_area and write_to_buffer

Hi,

this is not really an issue but I need advice: basically, I’m working with large TIF files (~100 000 x 100 000) and I need to extract patches from them as fast as possible, asynchronously from the main thread. So a naive approach would be this:

full_img = pyvips.Image.new_from_file('large.tif')
# called on many threads
def read_patch(coord):
    patch_img = full_img.extract_area(coord[0], coord[1], size[0], size[1])
    return np.ndarray(patch_img.write_to_memory(), ...)

But this is very slow, it takes ~23ms on average for a ~300x300 patch on my system, which is too much for my purposes. Reading larger areas first and then just slicing them for the patches was about equally slow, so I thought what slowed me down was mostly the decompression time from the TIF (I’m not exactly sure how it is compressed, tiled, etc.).

So I just converted the TIF to binary first and then re-did as above using the VIPS file format. And it still takes ~20ms on average for a ~300x300 patch. However, now reading larger areas is much faster, so I can get the speed down to ~3ms by reading 100 000 x 300 rows and then extracting from there.

I suspect that this is an issue with synchronization, these experiments are done with fairly recent hardware (as in last two years) and reading from a SSD, so I don’t believe the problem is the disk access speed or RAM/CPU/bus speeds.

Do you have any idea why it could be slow to read many patches from different threads ?

Note: I also tried experimenting with vips_cache() since I know I will be reading patches in regular intervals but the cached image always returned zero-values on subsequent write_to_memory() calls (it does work when called from main thread though, so I assume this is an issue with threading as well).

Finally, it’s not really an option for me to just keep the file in binary/VIPS format, because the original files are already >1GB and the binaries are >90GB which is hard to keep on a SSD. But if you have a suggestion for a compromise which is not too big but much faster to read for VIPS, it’s no problem to convert the files of course. Unfortunately, I don’t have much information about how the originals were created and I’m not really an expert on this, so I’m not even sure what the compression is etc.

[jcupitt]

It won’t scale well with threads: libtiff does the per-tile JPEG decompress inside a global mutex, unfortunately 😦

[MatthiasKohl]

Thank you again for the answer !