Question: OOM during composite and write large PNGs

I am trying to write large PNGs (its on a Kubernetes node with 8GB RAM).

• I seem to be able to write large PNGs (e.g. 70MB on disk) by taking a single “small” sample PNG and writing it multiple times to a large output image using .composite
• In my application, when i try to stream many PNGs and composite them onto a large output image it OOMs.

My question is about expected behavior and what is the pattern to do this … should it be possible to composite any number of “small” images streamed (i.e. loaded from S3 into memory one image at a time) and composite them onto a large output image and save to file? Assume these “small” images are much smaller in memory than 8GB.

I had assumed the the write would sequentially write out the PNG without holding the streamed images in memory but i must be mistaken? It is confusing to me right now that my first test that writes a single PNG works but my true test of writing out different images does not.

[jcupitt]

The disc threshold is the size of the uncompressed image, so width * height * bands * sizeof( element ).

You can put the unit after the number, so eg. “1gb”, “10mb”, “100kb”, etc.

There’s a page in the docs covering libvips image opening which gives some background:

https://www.libvips.org/API/current/How-it-opens-files.html

[Borda]

You can try tuning settings for VIPS_CONCURRENCY and VIPS_DISC_THRESHOLD

This looks useful! are both available also for python scripts? Could you please share what units for VIPS_DISC_THRESHOLD are used (MB or pixels)?