AWS Lambda Error

Hey there!

i try to get sharp with poppler support working on AWS lambda. In your installation instructions you have explained how to build sharp for lambda. This will download a pre build vips without the poppler support. Well, i have build my own docker image based on the lambci/lambda:build-nodejs8.10 image and installed all dependencies included all poppler stuff.

When i get into the docker container with the interactive shell and install sharp with the following command:

export PKG_CONFIG_PATH="${PKG_CONFIG_PATH}:${TARGET}/lib/pkgconfig"
npm install sharp

Sharp detects the globally installed vips:

info sharp Detected globally-installed libvips v8.6.4
info sharp Building from source via node-gyp

My little test-script to convert a PDF to PNG works great (see my test-repo):

LD_LIBRARY_PATH=${TARGET}/lib node index.js

Then i have created a simple lamda and bundled the include, lib, bin folder into it (builded in the custom docker container). Also installed sharp inside the container, placed in the deployment package in the compiled folder and use inside the lamda handler require('../compiled/sharp'); to get the precompiled sharp.

The deployed lambda crashes with the following error:

{"errorMessage":"libvips-cpp.so.42: cannot open shared object file: No such file or directory","errorType":"Error","stackTrace":["Object.Module._extensions..node (module.js:681:18)","Module.load (module.js:565:32)","tryModuleLoad (module.js:505:12)","Function.Module._load (module.js:497:3)","Module.require (module.js:596:17)","require (internal/module.js:11:18)","Object.<anonymous> (/var/task/compiled/sharp/lib/constructor.js:10:15)","Module._compile (module.js:652:30)","Object.Module._extensions..js (module.js:663:10)","Module.load (module.js:565:32)"]}

The bundled folders placed in the lambda task root (/var/task) and will be deployd. Regarding to this doc lamda sets the LD_LIBRARY_PATH env var to LAMBDA_TASK_ROOT/lib. The error occurs in the libvips-cpp.so.42 file, so i assume sharp find the libvips library.

Any suggestions i can try to get this working? Or any idea what the error means?

Many thanks!

[jweisman]

For those who stumble on this issue (including our future selves), we were able to resolve this using

• this Dockerfile for the build environment,
• this script to extract the dependencies for Imagemagick and the jp2 coder,
• and adding these environment variables to point vips to the Imagemagick files.

Not sure if there is an easier way, but this is what worked for us.

[jweisman]

Hi @lovell. I am having the same problem outlined here. I want to support JP2/JPEG2000 files in an AWS Lambda environment. So following the comment in #1489, I have compiled vips with ImageMagick support in a container based on the lambda:build-nodejs12.x image. Inside the container I npm install sharp and it recognized the globally-installed vips.

When I try to access a JPEG2000 file in the build container using the script below:

const sharp = require('sharp');

console.log(sharp.format.magick);
sharp('/tmp/test/balloon.jp2')
  .metadata()
  .then(console.log)
  .catch(console.error);

And I get a good response:

$ LD_LIBRARY_PATH=/opt/lib node test.js
{
  id: 'magick',
  input: { file: true, buffer: true, stream: true },
  output: { file: true, buffer: true, stream: true }
}
{
  format: 'magick',
  width: 2717,
  height: 3701,
  space: 'srgb',
  channels: 3,
  depth: 'uchar',
  isProgressive: false,
  pages: 1,
  hasProfile: false,
  hasAlpha: false
}

Then I used ldd nodejs/node_modules/sharp/build/Release/sharp.node to retrieve all of the dependent libraries and mount them as /opt/lib in the runtime container (lambci/lambda:nodejs12.x). Then I run the same script in the runtime specifying the LD_LIBRARY_PATH environment variable:

$ LD_LIBRARY_PATH=/opt/lib node test.js
{
  id: 'magick',
  input: { file: true, buffer: true, stream: true },
  output: { file: true, buffer: true, stream: true }
}
[Error: Input file contains unsupported image format]

I checked the dependencies of sharp.node and it seems that everything is the same in both environments:

$ LD_LIBRARY_PATH=/opt/lib ldd /var/task/node_modules/sharp/build/Release/sharp.node
	linux-vdso.so.1 (0x00007ffd9fbc1000)
	libvips-cpp.so.42 => /opt/lib/libvips-cpp.so.42 (0x00007f8f2aba2000)
	libvips.so.42 => /opt/lib/libvips.so.42 (0x00007f8f2a4de000)
	libgobject-2.0.so.0 => /opt/lib/libgobject-2.0.so.0 (0x00007f8f2a28b000)
	libglib-2.0.so.0 => /opt/lib/libglib-2.0.so.0 (0x00007f8f29f75000)
	libstdc++.so.6 => /opt/lib/libstdc++.so.6 (0x00007f8f29bf3000)
	libm.so.6 => /opt/lib/libm.so.6 (0x00007f8f298b3000)
	libgcc_s.so.1 => /opt/lib/libgcc_s.so.1 (0x00007f8f2969d000)
	libpthread.so.0 => /opt/lib/libpthread.so.0 (0x00007f8f2947f000)
	libc.so.6 => /opt/lib/libc.so.6 (0x00007f8f290d4000)
	libz.so.1 => /opt/lib/libz.so.1 (0x00007f8f28ebe000)
	libMagickCore.so.5 => /opt/lib/libMagickCore.so.5 (0x00007f8f28a10000)
	libpng15.so.15 => /opt/lib/libpng15.so.15 (0x00007f8f287e5000)
	libtiff.so.5 => /opt/lib/libtiff.so.5 (0x00007f8f28571000)
	libjpeg.so.62 => /opt/lib/libjpeg.so.62 (0x00007f8f2831f000)
	libgthread-2.0.so.0 => /opt/lib/libgthread-2.0.so.0 (0x00007f8f2811d000)
	libgmodule-2.0.so.0 => /opt/lib/libgmodule-2.0.so.0 (0x00007f8f27f19000)
	libexpat.so.1 => /opt/lib/libexpat.so.1 (0x00007f8f27ce7000)
	libgsf-1.so.114 => /opt/lib/libgsf-1.so.114 (0x00007f8f27aa5000)
	libxml2.so.2 => /opt/lib/libxml2.so.2 (0x00007f8f2774c000)
	libgif.so.4 => /opt/lib/libgif.so.4 (0x00007f8f27542000)
	librsvg-2.so.2 => /opt/lib/librsvg-2.so.2 (0x00007f8f2730c000)
	libgio-2.0.so.0 => /opt/lib/libgio-2.0.so.0 (0x00007f8f26f6f000)
	libgdk_pixbuf-2.0.so.0 => /opt/lib/libgdk_pixbuf-2.0.so.0 (0x00007f8f26d48000)
	libcairo.so.2 => /opt/lib/libcairo.so.2 (0x00007f8f26a22000)
	libopenslide.so.0 => /opt/lib/libopenslide.so.0 (0x00007f8f267f3000)
	libexif.so.12 => /opt/lib/libexif.so.12 (0x00007f8f265ae000)
	libpcre.so.1 => /opt/lib/libpcre.so.1 (0x00007f8f2634a000)
	libffi.so.6 => /opt/lib/libffi.so.6 (0x00007f8f26142000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f8f2b011000)
	libfontconfig.so.1 => /opt/lib/libfontconfig.so.1 (0x00007f8f25f05000)
	libfreetype.so.6 => /opt/lib/libfreetype.so.6 (0x00007f8f25c61000)
	libXext.so.6 => /opt/lib/libXext.so.6 (0x00007f8f25a4f000)
	libSM.so.6 => /opt/lib/libSM.so.6 (0x00007f8f25847000)
	libICE.so.6 => /opt/lib/libICE.so.6 (0x00007f8f2562a000)
	libX11.so.6 => /opt/lib/libX11.so.6 (0x00007f8f252ec000)
	libXt.so.6 => /opt/lib/libXt.so.6 (0x00007f8f25084000)
	liblzma.so.5 => /opt/lib/liblzma.so.5 (0x00007f8f24e5e000)
	libbz2.so.1 => /opt/lib/libbz2.so.1 (0x00007f8f24c4e000)
	libgomp.so.1 => /opt/lib/libgomp.so.1 (0x00007f8f24a21000)
	libltdl.so.7 => /opt/lib/libltdl.so.7 (0x00007f8f24817000)
	libjbig.so.2.0 => /opt/lib/libjbig.so.2.0 (0x00007f8f2460b000)
	libdl.so.2 => /opt/lib/libdl.so.2 (0x00007f8f24407000)
	libpangocairo-1.0.so.0 => /opt/lib/libpangocairo-1.0.so.0 (0x00007f8f241fa000)
	libpango-1.0.so.0 => /opt/lib/libpango-1.0.so.0 (0x00007f8f23faf000)
	libcroco-0.6.so.3 => /opt/lib/libcroco-0.6.so.3 (0x00007f8f23d74000)
	libselinux.so.1 => /opt/lib/libselinux.so.1 (0x00007f8f23b4d000)
	libresolv.so.2 => /opt/lib/libresolv.so.2 (0x00007f8f23937000)
	libmount.so.1 => /opt/lib/libmount.so.1 (0x00007f8f236e2000)
	libpixman-1.so.0 => /opt/lib/libpixman-1.so.0 (0x00007f8f2343c000)
	libEGL.so.1 => /opt/lib/libEGL.so.1 (0x00007f8f23228000)
	libxcb-shm.so.0 => /opt/lib/libxcb-shm.so.0 (0x00007f8f23025000)
	libxcb.so.1 => /opt/lib/libxcb.so.1 (0x00007f8f22dfd000)
	libxcb-render.so.0 => /opt/lib/libxcb-render.so.0 (0x00007f8f22bef000)
	libXrender.so.1 => /opt/lib/libXrender.so.1 (0x00007f8f229e5000)
	libGL.so.1 => /opt/lib/libGL.so.1 (0x00007f8f22759000)
	librt.so.1 => /opt/lib/librt.so.1 (0x00007f8f22551000)
	libsqlite3.so.0 => /opt/lib/libsqlite3.so.0 (0x00007f8f2229c000)
	libopenjpeg.so.1 => /opt/lib/libopenjpeg.so.1 (0x00007f8f22079000)
	libuuid.so.1 => /opt/lib/libuuid.so.1 (0x00007f8f21e74000)
	libpangoft2-1.0.so.0 => /opt/lib/libpangoft2-1.0.so.0 (0x00007f8f21c5f000)
	libthai.so.0 => /opt/lib/libthai.so.0 (0x00007f8f21a53000)
	libharfbuzz.so.0 => /opt/lib/libharfbuzz.so.0 (0x00007f8f217ef000)
	libblkid.so.1 => /opt/lib/libblkid.so.1 (0x00007f8f215a5000)
	libGLdispatch.so.0 => /opt/lib/libGLdispatch.so.0 (0x00007f8f212ef000)
	libXau.so.6 => /opt/lib/libXau.so.6 (0x00007f8f210eb000)
	libGLX.so.0 => /opt/lib/libGLX.so.0 (0x00007f8f20eba000)
	libgraphite2.so.3 => /opt/lib/libgraphite2.so.3 (0x00007f8f20c8e000)

Note that if I install ImageMagick in the lambda:build-nodejs12.x image, then I can successfully process my jp2 file. So it appears that ldd is not giving me all of the sharp/vips dependencies and that sharp is relying on additional libraries at runtime.

Any help would be greatly appreciated.