Enhancement: verify current+vendored platform match during npm rebuild

The pre-compiled dependencies placed in node_modules/sharp/vendor are platform-specific. Some people, including those using Meteor, use npm rebuild for cross-platform support so sharp should verify the current platform matches the vendored platform.

Whilst we could place the vendor directory under build and get the deletion of this via npm rebuild for “free”, this would involve re-downloading ~7MB of files that are already present.

Instead using platform-specific directories below vendor would make more sense, e.g. node_modules/sharp/vendor/darwin, node_modules/sharp/vendor/linux and node_modules/sharp/vendor/win32.

See also #637

[lovell]

Please can you open a new issue for this. I’ll be able to take a look in a couple of weeks.

[csKangGdcomm]

@lovell I faced similar problem like @mutaphysis mentioned.

on my dev machine (macOS) it works well. but on the production environment(AWS linux), it is not working…

I am using meteor and this library ( both of them are latest version )

And I am using elastic beenstalk to delploy then

I got error like below at eb-activity.log

> mup-myfirstfoundueapp@118.0.0 preinstall /tmp/deployment/application
  > cd programs/server && npm install
  
  
  > fibers@2.0.0 install /tmp/deployment/application/programs/server/node_modules/fibers
  > node build.js || nodejs build.js
  
  `linux-x64-48` exists; testing
  Binary is fine; exiting
  
  > meteor-dev-bundle@ install /tmp/deployment/application/programs/server
  > node npm-rebuild.js
  
  
  > bcrypt@1.0.3 install /tmp/deployment/application/programs/server/npm/node_modules/bcrypt
  > node-pre-gyp install --fallback-to-build
  
  [bcrypt] Success: "/tmp/deployment/application/programs/server/npm/node_modules/bcrypt/lib/binding/bcrypt_lib.node" is installed via remote
  
  > sharp@0.20.5 install /tmp/deployment/application/programs/server/npm/node_modules/sharp
  > (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)
  
  info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.6.1/libvips-8.6.1-linux-x64.tar.gz
  fs.js:932
    return binding.mkdir(pathModule._makeLong(path),
                   ^
  
  Error: EEXIST: file already exists, mkdir '/tmp/deployment/application/programs/server/npm/node_modules/sharp/vendor'
      at Error (native)
      at Object.fs.mkdirSync (fs.js:932:18)
      at WriteStream.<anonymous> (/tmp/deployment/application/programs/server/npm/node_modules/sharp/install/libvips.js:61:10)
      at emitNone (events.js:86:13)
      at WriteStream.emit (events.js:185:7)
      at fs.js:2059:14
      at FSReqWrap.oncomplete (fs.js:123:15)
  make: Entering directory `/tmp/deployment/application/programs/server/npm/node_modules/sharp/build'
    TOUCH Release/obj.target/libvips-cpp.stamp
    CXX(target) Release/obj.target/sharp/src/common.o
    CXX(target) Release/obj.target/sharp/src/metadata.o
    CXX(target) Release/obj.target/sharp/src/stats.o
    CXX(target) Release/obj.target/sharp/src/operations.o
    CXX(target) Release/obj.target/sharp/src/pipeline.o
    CXX(target) Release/obj.target/sharp/src/sharp.o
    CXX(target) Release/obj.target/sharp/src/utilities.o
    SOLINK_MODULE(target) Release/obj.target/sharp.node
  g++: error: ../vendor/lib/libvips-cpp.so: No such file or directory
  g++: error: ../vendor/lib/libvips.so: No such file or directory
  g++: error: ../vendor/lib/libglib-2.0.so: No such file or directory
  g++: error: ../vendor/lib/libgobject-2.0.so: No such file or directory
  g++: error: ../vendor/lib/libcairo.so: No such file or directory
  g++: error: ../vendor/lib/libcroco-0.6.so: No such file or directory
  g++: error: ../vendor/lib/libexif.so: No such file or directory
  g++: error: ../vendor/lib/libffi.so: No such file or directory
  g++: error: ../vendor/lib/libfontconfig.so: No such file or directory
  g++: error: ../vendor/lib/libfreetype.so: No such file or directory
  g++: error: ../vendor/lib/libgdk_pixbuf-2.0.so: No such file or directory
  g++: error: ../vendor/lib/libgif.so: No such file or directory
  g++: error: ../vendor/lib/libgio-2.0.so: No such file or directory
  g++: error: ../vendor/lib/libgmodule-2.0.so: No such file or directory
  g++: error: ../vendor/lib/libgsf-1.so: No such file or directory
  g++: error: ../vendor/lib/libgthread-2.0.so: No such file or directory
  g++: error: ../vendor/lib/libharfbuzz.so: No such file or directory
  g++: error: ../vendor/lib/libjpeg.so: No such file or directory
  g++: error: ../vendor/lib/liblcms2.so: No such file or directory
  g++: error: ../vendor/lib/liborc-0.4.so: No such file or directory
  g++: error: ../vendor/lib/libpango-1.0.so: No such file or directory
  g++: error: ../vendor/lib/libpangocairo-1.0.so: No such file or directory
  g++: error: ../vendor/lib/libpangoft2-1.0.so: No such file or directory
  g++: error: ../vendor/lib/libpixman-1.so: No such file or directory
  g++: error: ../vendor/lib/libpng.so: No such file or directory
  g++: error: ../vendor/lib/librsvg-2.so: No such file or directory
  g++: error: ../vendor/lib/libtiff.so: No such file or directory
  g++: error: ../vendor/lib/libwebp.so: No such file or directory
  g++: error: ../vendor/lib/libxml2.so: No such file or directory
  g++: error: ../vendor/lib/libz.so: No such file or directory
  make: *** [Release/obj.target/sharp.node] Error 1
  make: Leaving directory `/tmp/deployment/application/programs/server/npm/node_modules/sharp/build'
  gyp ERR! build error 
  gyp ERR! stack Error: `make` failed with exit code: 2
  gyp ERR! stack     at ChildProcess.onExit (/opt/elasticbeanstalk/node-install/node-v6.12.2-linux-x64/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:276:23)
  gyp ERR! stack     at emitTwo (events.js:106:13)
  gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
  gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:219:12)
  gyp ERR! System Linux 4.9.85-38.58.amzn1.x86_64
  gyp ERR! command "/opt/elasticbeanstalk/node-install/node-v6.12.2-linux-x64/bin/node" "/opt/elasticbeanstalk/node-install/node-v6.12.2-linux-x64/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
  gyp ERR! cwd /tmp/deployment/application/programs/server/npm/node_modules/sharp
  gyp ERR! node -v v6.12.2
  gyp ERR! node-gyp -v v3.4.0
  gyp ERR! not ok 
  
  npm ERR! Linux 4.9.85-38.58.amzn1.x86_64
  npm ERR! argv "/opt/elasticbeanstalk/node-install/node-v6.12.2-linux-x64/bin/node" "/opt/elasticbeanstalk/node-install/node-v6.12.2-linux-x64/bin/npm" "rebuild" "--update-binary"
  npm ERR! node v6.12.2
  npm ERR! npm  v3.10.10
  npm ERR! code ELIFECYCLE
  npm ERR! sharp@0.20.5 install: `(node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)`
  npm ERR! Exit status 1
  npm ERR! 
  npm ERR! Failed at the sharp@0.20.5 install script '(node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)'.
  npm ERR! Make sure you have the latest version of node.js and npm installed.
  npm ERR! If you do, this is most likely a problem with the sharp package,
  npm ERR! not with npm itself.
  npm ERR! Tell the author that this fails on your system:
  npm ERR!     (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)
  npm ERR! You can get information on how to open an issue for this project with:
  npm ERR!     npm bugs sharp
  npm ERR! Or if that isn't available, you can get their info via:
  npm ERR!     npm owner ls sharp
  npm ERR! There is likely additional logging output above.
  
  npm ERR! Please include the following file with any support request:
  npm ERR!     /tmp/deployment/application/programs/server/npm/npm-debug.log
  
  npm ERR! Linux 4.9.85-38.58.amzn1.x86_64
  npm ERR! argv "/opt/elasticbeanstalk/node-install/node-v6.12.2-linux-x64/bin/node" "/opt/elasticbeanstalk/node-install/node-v6.12.2-linux-x64/bin/npm" "install"
  npm ERR! node v6.12.2
  npm ERR! npm  v3.10.10
  npm ERR! code ELIFECYCLE
  npm ERR! meteor-dev-bundle@ install: `node npm-rebuild.js`
  npm ERR! Exit status 1
  npm ERR! 
  npm ERR! Failed at the meteor-dev-bundle@ install script 'node npm-rebuild.js'.
  npm ERR! Make sure you have the latest version of node.js and npm installed.
  npm ERR! If you do, this is most likely a problem with the meteor-dev-bundle package,
  npm ERR! not with npm itself.
  npm ERR! Tell the author that this fails on your system:
  npm ERR!     node npm-rebuild.js
  npm ERR! You can get information on how to open an issue for this project with:
  npm ERR!     npm bugs meteor-dev-bundle
  npm ERR! Or if that isn't available, you can get their info via:
  npm ERR!     npm owner ls meteor-dev-bundle
  npm ERR! There is likely additional logging output above.
  
  npm ERR! Please include the following file with any support request:
  npm ERR!     /tmp/deployment/application/programs/server/npm-debug.log
  
  npm WARN mup-myfirstfoundueapp@118.0.0 No repository field.
  npm WARN mup-myfirstfoundueapp@118.0.0 No license field.
  npm ERR! Linux 4.9.85-38.58.amzn1.x86_64
  npm ERR! argv "/opt/elasticbeanstalk/node-install/node-v6.12.2-linux-x64/bin/node" "/opt/elasticbeanstalk/node-install/node-v6.12.2-linux-x64/bin/npm" "--production" "install"
  npm ERR! node v6.12.2
  npm ERR! npm  v3.10.10
  npm ERR! code ELIFECYCLE
  npm ERR! mup-myfirstfoundueapp@118.0.0 preinstall: `cd programs/server && npm install`
  npm ERR! Exit status 1
  npm ERR! 
  npm ERR! Failed at the mup-myfirstfoundueapp@118.0.0 preinstall script 'cd programs/server && npm install'.
  npm ERR! Make sure you have the latest version of node.js and npm installed.
  npm ERR! If you do, this is most likely a problem with the mup-myfirstfoundueapp package,
  npm ERR! not with npm itself.
  npm ERR! Tell the author that this fails on your system:
  npm ERR!     cd programs/server && npm install
  npm ERR! You can get information on how to open an issue for this project with:
  npm ERR!     npm bugs mup-myfirstfoundueapp
  npm ERR! Or if that isn't available, you can get their info via:
  npm ERR!     npm owner ls mup-myfirstfoundueapp
  npm ERR! There is likely additional logging output above.
  npm WARN Local package.json exists, but node_modules missing, did you mean to install?
  
  npm ERR! Please include the following file with any support request:
  npm ERR!     /tmp/deployment/application/npm-debug.log
  Running npm install:  /opt/elasticbeanstalk/node-install/node-v6.12.2-linux-x64/bin/npm
  Setting npm config jobs to 1
  npm config jobs set to 1
  Running npm with --production flag
  Failed to run npm install. Snapshot logs for more details.
  Traceback (most recent call last):
    File "/opt/elasticbeanstalk/containerfiles/ebnode.py", line 695, in <module>
      main()
    File "/opt/elasticbeanstalk/containerfiles/ebnode.py", line 677, in main
      node_version_manager.run_npm_install(options.app_path)
    File "/opt/elasticbeanstalk/containerfiles/ebnode.py", line 136, in run_npm_install
      self.npm_install(bin_path, self.config_manager.get_container_config('app_staging_dir'))
    File "/opt/elasticbeanstalk/containerfiles/ebnode.py", line 180, in npm_install
      raise e
  subprocess.CalledProcessError: Command '['/opt/elasticbeanstalk/node-install/node-v6.12.2-linux-x64/bin/npm', '--production', 'install']' returned non-zero exit status 1 (ElasticBeanstalk::ExternalInvocationError)
caused by: + /opt/elasticbeanstalk/containerfiles/ebnode.py --action npm-install

I also tried the npm clean ... on node_modules/sharps it still doesn’t work