Possible TargetCustom Leak

Hello,

I am using VIPS on a ASP.NET Core application, in order to avoid frequent allocation of memory, I am using TargetCustom to directly write the output to a PipeWriter that is attached to the HTTP response stream. This works fine on all normal circumstance. However, when the requests to the app increases and the client starts to disconnect abruptly, there seem to be some issue while disposing TargetCustom.

NetVips: 1.2.4
NetVips.Native: v8.10

Here is the error message I am getting: Process terminated. A callback was made on a garbage collected delegate of type 'NetVips!NetVips.Internal.VipsSourceCustom+ReadSignal::Invoke'.

This is rough idea of how my code looks:

            PipeWriter writer; // piped to the HTTP response stream
            ...
            using var target = new TargetCustom();
            target.OnWrite += (buffer, length) =>
            {
                var span = new ReadOnlySpan<byte>(buffer, 0, length);
                writer.Write(span);
                return length;
            };
            target.OnFinish += () => writer.Complete();
            _image.WriteToTarget(target

I tried to manually detach the OnWrite and OnFinish delegate, and the error still remains. Is there any work around for this?

Secondly, the same application also seem to have some other issue, occasionally it crashes with: Segmentation fault (core dumped)

on a windows server it produces: The program has exited with code -1073741819 (0xc0000005) 'Access violation'.

Unfortunately I cannot find any other stack trace. Since it is a web app, each request can independently resize images in parallel, I am afraid this may be related to multithreading. The method ConcurrencySet is not used, leaving VIPS to automatically choose based on the number of CPUs. The issue also seems to happen when the main memory is overwhelmed, and is not easily reproduced.

When the app is first initialized, I am statically setting these:

            NetVips.Init();
            NetVips.CacheSetMax(0);
            NetVips.CacheSetMaxFiles(0);
            NetVips.CacheSetMaxMem(0);
            NetVips.CacheSetTrace(false);
            NetVips.LeakSet(true);

Thank you so much for taking the time to look into the issues.

-Jerome

[kleisauke]

Sorry for the delay, I should spend more time on replying to issues.

Great to hear that the crashes have been resolved now! I’ve no experience with Kubernetes, but can provide some performance/security tips for image processing with libvips. These tips are mainly based on my experience with images.weserv.nl, which currently processes 7 million (7×10⁶) images per hour.

• Avoid using ThumbnailImage, if possible. It can’t do any of the shrink-on-load tricks, so it’s much, much slower.
• Consider using a different memory allocator such as jemalloc to prevent memory fragmentation in long-running, multi-threaded, glibc-based Linux processes. See https://github.com/lovell/sharp/issues/955 to learn more about memory allocation and why the choice of a memory allocator is important.
• The libvips’ operation cache is not useful for image proxies (i.e. processing many different images). Consider to disable this.
• Only enable the loading libraries you really need. For example, libvips could open microscope images, which most websites won’t need.
• Always sanity check image dimensions to protect you from decompression bombs like those described at https://www.bamsoftware.com/hacks/deflate.html.

[jerome-zenfolio]

Thank you Kleisauke & congrats on the 2.0 release!