NewFromBuffer causes GLib-ERROR with x86 libvips

Hi @kleisauke, thanks a ton for this libvips wrapper. I’ve learned a lot from reading your correspondence with @jcupitt.

Based on the advice in https://images.guide/, I’m currently reimplementing an image optimization pipeline with libvips at the core. Unfortunately, the larger program that this is a part of references other native x86 assemblies so for now I’m working with x86 libvips.

The x86 3-4GB memory limit seems to be causing an error that looks like (NetVips:17212): GLib-ERROR **: 11:19:21.666: /data/mxe/tmp-glib-i686-w64-mingw32.shared/glib-2.59.0/glib/gmem.c:105: failed to allocate 39493103 bytes.

I reproduced this issue in an isolated project and included large and small test images. https://github.com/jjonesrs/NetVipsLeakTest

Here it is inline:

byte[] imageBytes = File.ReadAllBytes(@"large-test-image.png");
//byte[] imageBytes = File.ReadAllBytes(@"small-test-image.jpg");

Base.LeakSet(1);

Operation.VipsCacheSetMax(0);
Operation.VipsCacheSetMaxFiles(0);

for (int i = 0; i < 1000000; i++)
{
    using (Image img = Image.NewFromBuffer(imageBytes))
    {
        Console.WriteLine(img);
    }
}

The above small snippet might look unrealistic, but it intentionally doesn’t do any more than call Image.NewFromBuffer() on the source image to serve as a test of loading from a buffer in rapid succession. Loading from a file understandably does not exhibit the same memory patterns, but I’d highly prefer not to pull images from S3, save them to local disk, then risk file lock contention on delete.

I would expect the using ... statement to call Dispose() and signal to libvips to free the native memory at some point before the GLib crash, but it doesn’t seem to have any effect.

I don’t think that NetVips or libvips is strictly leaking, but it looks like maybe libvips isn’t freeing images fast enough to keep up with the low x86 memory limit. Testing under x64, I can’t reproduce the GLib error, but I do observe (in my opinion) high memory usage of 2-4GB.

I’ve included two memory profiles from JetBrains dotMemory leading up to the crash.

large-test-image.png:

small-test-image.jpg:

[kleisauke]

I’ve just released NetVips 1.1.0-rc1 and NetVips.Native 8.8.0-rc1 (which contains the pre-compiled libvips 8.8.0-rc1 binaries for Linux, macOS and Windows) on NuGet. This release should fix the GC issue described above.

If you would like to see what’s new in libvips 8.8, please visit the the release notes of libvips: https://libvips.github.io/libvips/2019/04/22/What’s-new-in-8.8.html

I’m sorry it took so long to release this. By separating the pre-compiled libraries from the binding itself, it’s now easier to release a new version.

[jcupitt]

I had a quick go in Python:

import sys
import pyvips

pyvips.cache_set_max(0)

buf = open(sys.argv[1], "rb").read()

for i in range(0, 100000):
    if i % 100 == 0:
        print("loop {} ...".format(i))
        
    image = pyvips.Image.new_from_buffer(buf, "")

And I see a steady 45mb of RES used.

If you are seeing 2 - 4gb, I’d guess the GC isn’t running often enough.