Segmentation fault (core dumped) from pyvips

I’m trying to resize a large image to a smaller width with a limited memory available 256MB (due to AWS lambda configuration). I used the code like this:

import pyvips

image = pyvips.Image.new_from_file(src_file, access='sequential')
image = image.resize(0.8/1.0)
image.write_to_file(dst_file)

In my local machine I’m testing it by putting a limit on Python VM using code like:

resource.setrlimit(resource.RLIMIT_AS, (maxsize, hard))

when I set the maxsize to 256MB, I get Segmentation fault (core dumped) error. I suppose this is coming from the vips C library. So, I have two questions:

1. Why the memory issue in vips library (assuming there is one) is not raised as MemoryError for python to handle gracefully?
2. How can I use pyvips to resize a very big image with relatively small memory environment such as AWS lambda? [I can sacrifice CPU performance but I need the memory performance]

[jcupitt]

Sorry, I never answered your first question.

Out of memory errors are generally impossible to handle gracefully. Suppose memory is full and a fault is raised. What is going to handle the error? If no more memory can be used, no fault handler can execute. Like almost all software, libvips treats memory as an infinite resource and relies on the environment to manage it.

libvips does check large single allocations (for example, loading a 10gb image into memory), but does not check small ones, the reasoning being that, if a small allocation fails, the process is as good as dead.

[jcupitt]

I tried this program:

#!/usr/bin/python 

import sys
import pyvips 

pyvips.leak_set(True)

image = pyvips.Image.new_from_file(sys.argv[1], access='sequential')
image = image.resize(0.8 / 1.0)
image.write_to_file(sys.argv[2])

leak_set makes libvips run a little slower, but it will check for leaks and report high-water memory usage. With a 10k x 10k pixel RGB image I see:

john@kiwi:~/try$ ./resize.py ~/pics/wtc.jpg x.jpg
memory: high-water mark 53.48 MB

If I change the program like this:

#!/usr/bin/python 

import sys
import pyvips 

pyvips.cache_set_max(0)
pyvips.leak_set(True)

image = pyvips.Image.thumbnail(sys.argv[1], 8000)
image.write_to_file(sys.argv[2])

I see:

john@kiwi:~/try$ VIPS_CONCURRENCY=1 ./resize.py ~/pics/wtc.jpg x.jpg
memory: high-water mark 33.82 MB

That’s running with ulimit -m 256000, which might be equivalent to your settings. I couldn’t get it to crash.