How to create a tiled TIFF without requiring to keep the whole image in memory?

Hi, I have the following problem: I want to create a tiled TIFF, which is rather large (>200000x100000) such that it doesn’t fit into RAM memory. I am creating individual tiles (1024x1024, numpy arrays) dynamically in python and want to add them to the resulting TIFF. Is that possible with vips, without requiring to keep the whole image in memory?

I’ve tried lots of different approaches with vips, however, all of them resulted in allocating lots of RAM memory.

The following sourcecode is one example of an approach that I’ve tried.

1. create tiles, and write them to disk: (create_tiles() function)
2. fuse all the tiles together to a TIFF (combine_tiles_to_wsi() function)

import cv2
import numpy as np
import pyvips


def create_tiles():
    w = 200
    h = 100

    im = np.zeros((1024, 1024, 3))
    for i in range(h*w):
        cv2.imwrite('tmp/tiles/{:07d}.jpg'.format(i), im)


def combine_tiles_to_wsi():
    w = 200
    h = 100

    print('creating images from array')
    vimgs = [pyvips.Image.new_from_file('tmp/tiles/{:07d}.jpg'.format(i), access='sequential') for i in range(h*w)]

    print('joining arrays')
    vimg = pyvips.Image.arrayjoin(vimgs, across=w)
    print('writing to files')
    vimg.write_to_file('test.tiff', tile=True, tile_width=1024, tile_height=1024,
                       pyramid=True,
                       compression='jpeg',
                       )

if __name__ == "__main__":
    create_tiles()
    combine_tiles_to_wsi()

When I run the code above vips starts eating up my RAM memory during the call to “vimg.write_to_file”. At least more than 30GB of RAM are consumed by the program. (after that I had to kill it) I am using “pyvips==2.1.11” with “vips-8.7.0-Mon Sep 10 13:35:49 UTC 2018” on Ubuntu. I have also tried setting pyvips.cache_set_max(0) without success

[jonasteuwen]

+1 for the feature to write a tiff tile by tile without keeping the whole thing in memory.

[cookmscott]

Hi folks! I checked through the 8.10 release notes and nothing is jumping out to me for a better way to append tiles (aka individual images) to a TIFF without creating intermediaries and racking up memory usage for vips. Is above still the best option or did I miss a new feature? 😃

As an additional option… if I save my intermediary files as TIFFs with equal compression, can I join/write them without reprocessing or reading the entire image into memory?

(sorry for hijacking the thread, some folks don’t like comments on closed issues but hoping this will help close the loop!)