Lambda storage space shrinking

We are using Sharp in an AWS Lambda function, and we have an issue with storage space (of all things…). Function downloads an image from S3, places it inside tmp directory. After that:

var image = sharp('/tmp/imageName.png');
image.rotate();
image.toFormat('PNG').toBuffer(function(err, data) {
    // here we check storage space available and files inside the /tmp dir
    // some more stuff
})

With one 18MB PNG image we observed that, after outputting to buffer, the storage space available had dropped from 496MB (514MB available to Lambda minus 18MB for the image we placed inside tmp) to 142MB. This is a problem, because Lambda may reuse the previously used container, in which case the available storage remains at 160MB (142 + 18MB for the image, that we delete before exiting function), and if we upload the same image twice in a row, and the same container gets used, the function runs out of free space on disk and dies.

Question is simple - does Sharp and / or it’s underlying libraries, create any temporary files, that may not get deleted, or could this just be bad memory management, causing a swap file to be filled up?

I’m still waiting on a response from AWS, because I do not know if a swap file even exists inside the Lambda instance, so I’m not sure what might be causing this sudden spike in storage space usage. Also, other images don’t seem to be behaving this way - a 9MB PNG file doesn’t cause any spikes in disk space usage from my observations.

Sadly I cannot share the image itself.

I thank you for any guidance in advance.

[jcupitt]

The thumbnail operation gets around this problem by copying the shrunk image to a memory buffer before the rotate. You can see the logic here:

https://github.com/jcupitt/libvips/blob/master/libvips/resample/thumbnail.c#L479

The idea is that the shrunk image will usually be smaller (!!), so copying that to memory (rather than the source image) saves some RAM. It also drops latency, since you can hide the decode inside processing.

[KrissKulins]

Thank you @lovell , setting the VIPS_DISC_THRESHOLD fixed the issue.

There is one thing to add though - at first I tried it with sequentialRead() and that worked most of the time, however one image did fail.

This image, taken by a Samsung Galaxy S6, fails at toFormat with the following Lambda logs: 2017-02-10T16:17:42.506Z 7271c281-efac-11e6-9c2e-c52e7d71286e [Error: VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 VipsJpeg: out of order read at line 2988 ] A guy here had a similiar problem https://github.com/jcupitt/libvips/issues/261, only with libvips.

This is not an issue for me anymore, because setting environment variable fixed it, so this is more of an FYI.