Discussion around uws dependency in socket.io 2.x

Hi, I was doing a check of my node_modules folder last week and came across an issue around how socket.io uses the uws package as a dependency. I noticed that uws is an optional dependency with engine.io as of socket.io 2.0+ release notes and wanted to bring up a few things I noticed about it. I’ll write down how I discovered this, but because its a large post, here’s a TLDR to start with:

• The uws package on npm version 0.14.5 packages 10 binaries that are between 370KB and 1.5MB in size. These include binaries for other platforms and node js module versions. This causes the uws package to be ~10MB in size with most of it being useless binary packages. Even if you build uws through node-gyp on your own machine, you will still have these binaries when installing through npm.
• Unlike other npm packages that provide prebuilt binaries (like node-sass or node-serialport: https://github.com/sass/node-sass/releases , https://github.com/EmergingTechnologyAdvisors/node-serialport/releases ), uws does not separate out its platform dependent releases
• I worked on a fix which allows binaries to be prebuilt through Travis CI and Appveyor and uses Github for releases; as shown here: https://github.com/Flux159/uWebSockets/releases and https://www.npmjs.com/package/uws-light When I did this, I noticed that the linux binaries that are packaged with uws version 0.14.5 were about 0.5MB larger than the binaries that Travis was building. This is a pretty large discrepancy itself since I was simply using uws’s default Makefile to build. My builds are open and can be seen here: https://travis-ci.org/Flux159/uWebSockets/builds and here: https://ci.appveyor.com/project/Flux159/uwebsockets/history ; I believe that the maintainer of uws makes their own builds manually prior to publishing the npm package - there seems to be no open CI or CD system in place for uws
• uws’s package.json has an install script that will never fail: node-gyp rebuild > build_log.txt 2>&1 || exit 0 Since this redirects stderr to stdout and also forces an exit 0, npm will always think that the installation was successful. This means that even if I do something like break node-gyp by setting the python config to a non-existent python: npm config set python /usr/bin/p && npm install uws, uws will still be installed “successfully” with all the prebuilt binaries.

The long post:

To preface, I did bring this up with the uws repo first, as shown here: https://github.com/uNetworking/uWebSockets/issues/626 and here: https://github.com/uNetworking/bindings/pull/30, however the maintainer of the repo, seems to have no intention of fixing or even having a discussion around the issue.

Socket.io 2.x uses uws as an “optional” dependency. When doing some research into why my node_modules folder was ballooning in size, I noticed that uws was the 3rd largest package installed.

> du -sm node_modules/* | sort -rn | head -n 20
29	node_modules/monaco-editor
24	node_modules/nyc
11	node_modules/uws
…

Those sizes are in MB. (Before anyone comments on monaco-editor or nyc, I did make github issues/comments for those projects as well).

Checking the dependency tree using npm ls, I found that Socket.io was using uws as a dependency (technically an “optional” dependency, but will cover why its not necessarily optional below).

> npm ls uws
└─┬ socket.io@2.0.3
  └─┬ engine.io@3.1.0
    └── uws@0.14.5 

Wanting to see why uws was producing such a large npm package, I checked its directory:

> ls -l node_modules/uws/
…
-rwxr-xr-x   1 username  staff   377568 Apr 26 15:36 uws_darwin_46.node
-rwxr-xr-x   1 username  staff   377568 Apr 26 15:36 uws_darwin_47.node
-rwxr-xr-x   1 username  staff   377560 Apr 26 15:37 uws_darwin_48.node
-rwxr-xr-x   1 username  staff   377616 Apr 26 15:37 uws_darwin_51.node
-rwxr-xr-x   1 username  staff   377616 Jun 12 17:13 uws_darwin_57.node
-rwxr-xr-x   1 username  staff  1563976 Apr 26 15:11 uws_linux_46.node
-rwxr-xr-x   1 username  staff  1563976 Apr 26 15:11 uws_linux_47.node
-rwxr-xr-x   1 username  staff  1563976 Apr 26 15:11 uws_linux_48.node
-rwxr-xr-x   1 username  staff  1563976 Apr 26 15:12 uws_linux_51.node
-rwx------   1 username  staff   641024 Apr 26 15:17 uws_win32_48.node
-rwx------   1 username  staff   641536 Apr 26 15:18 uws_win32_51.node

Above, uws is packaging linux, windows, and other Mac OS X prebuilt binaries into its npm package. uws_darwin_57.node is the package that was built on my own system (using Node.js 8.x).

A couple of issues here: First, uws is packaging unnecessary binaries into its npm package - there’s no reason to have multiple platforms (“process.platform” in node) and multiple module versions (“process.versions.modules” in node) of binaries installed for one machine. This is causing the uws folder to be around 9MB larger than it needs to be. This really shouldn’t be an issue since there are well known solutions to the problem of having prebuilt binaries with npm - specifically use Github releases, AWS S3, or any other storage solution to upload binaries to, then download the appropriate binary at install time. I mentioned this to uws’s maintainer here: https://github.com/uNetworking/uWebSockets/issues/626

Since uws’s maintainer had no intention of fixing the unnecessary binary issue, I decided to make some builds using Travis CI and Appveyor using a fork of the uWebSockets repo. My fork adds an .appveyor.yml and travis.yml along with changing the install script to download from GitHub releases (and auto publishes to NPM on successful build): https://github.com/Flux159/uWebSockets https://www.npmjs.com/package/uws-light https://travis-ci.org/Flux159/uWebSockets/builds https://ci.appveyor.com/project/Flux159/uwebsockets/history Note that I do intend for anyone else to use those builds at the moment - the install script hasn’t been fully tested on all platforms (only OS X) and it was made in a very quick way.

This is where I found two other issues with uws’s npm package - the binaries that Travis CI was building for linux were significantly smaller than those in uws version 0.14.5 (1MB vs 1.5MB). I asked about this in issue 626 and received no response.

The last issue I found was with uws’s package.json install script. It was using this command:

node-gyp rebuild > build_log.txt 2>&1 || exit 0

That command will never fail since it is redirecting stderr to stdout and always exiting 0. This means that uws will still be installed “successfully” with all the prebuilt binaries even if node-gyp fails. There’s a partially reasonable reason for this if you’re trying to install uws directly from npm, but when installing engine.io this means that uws will always be installed regardless of whether its being used or not.

Again, I tried to submit a pull request to uws’s maintainer about a fix for the original issue here: https://github.com/uNetworking/bindings/pull/30 yet the pull request was rejected without (in my opinion) sufficient reasoning.

I wanted to open this issue to discuss moving engine.io’s dependency on uws to a different npm package / build due to the issues above (note I don’t mean moving off of the C++ based uWebSockets, I mean moving to a different npm build of uws). Whether this is a package maintained by socket.io or someone else, that could be discussed as well (I’m purposely not recommending using my npm package / build in fairness and since I haven’t done thorough testing on it).

Ping @darrachequesne

[ghost]

@Flux159 The reason I “don’t listen to criticism” is because your criticism is in direct opposition to other users criticism. You see anything popular will always result in some pleased users and some displeased users. You simply cannot cater for everyone. If 10mb of disk space is such a major deal for you, then maybe ws is the better choice for you. If you look at the very early issue reports of uws you will see this very topic being discussed in many different ways. Some users are opposed to binaries altogether, some users need node-gyp builds, some users just want things to work. I try and be open to all criticism and I try and balance it so that 90% are pelased and 10% are displeased. If you look at the download stats of uws and look at how the industry is using it (there are some very high-profile users of it), they seem to be perfectly pleased with it overall. If you really care that much about 10mb of disk space then you could really just go into the uws folder and delete those binaries you won’t need. I have reported this issue to NPM (about being able to upload per-platform binaries) but they refuse to listen. Again, I cannot change something as fundamental as the installation of something this well-used just because 3 users feel 10mb of disk space is too heavy. uws makes heavy use of C++ templates and prioritizes performance over disk space. This is why, for instance, I build the Linux binaries statically against libc++ - this has drawbacks such as increased disk space usage but also makes the lib work out of the box on virtually any Linux box. Pros and cons.

[STRML]

See https://github.com/uNetworking/uWebSockets-bindings