Diff does not contain packages if npm-shrinkwrap.json is missing "resolved" fields

I just had my first package update sequence go badly for me. As usual, I had done an npm install --save-dev (in this case, the “sinon-chai” package), confirmed the package was working as I wanted, and then run npm shrinkwrap --dev followed by shrinkpack. However, despite my package.json having been updated by the install, shrinkpack didn’t list any diff’d packages. I went back and forth on this for a while, and finally figured out what was going on.

I already knew that Shrinkpack relies on rewriting the “resolved” field in npm-shrinkpack.json to point to the local files under./node_shrinkwrap. However, it appears that on occasion, npm shrinkwrap does not actually write out a “resolved” field for some packages. In my case, the relevant shrinkwrap entry was:

    "sinon-chai": {
      "version": "2.8.0",
      "from": "sinon-chai@latest"
    }

Because shrinkwrap didn’t supply a “resolved” field, Shrinkpack didn’t see anything that needed to be included.

I tried a few times to get shrinkwrap to actually write out that field. There were a couple vaguely-related NPM issues, such as https://github.com/npm/npm/issues/3581 , which suggest this is a sorta-known issue with no real obvious solution. One suggestion was to run npm clear cache, which I did but didn’t help.

Ultimately, I tried running npm view sinon-chai to see the actual tarball URL, then manually added that as a “resolved” field to npm-shrinkwrap.json. When I reran Shrinkpack, it correctly indicated:

+ sinon-chai@2.8.0
shrinkpack +1 -0

Now, obviously Shrinkpack can’t do anything to force NPM install to write out npm-shrinkwrap.json properly. However, could it maybe do a little extra work to try to track down a valid resolution URL or something? For example, I note that if I look at my /node_modules/sinon-chai/package.json, it has a “dist” field with the same correct tarball URL. Maybe if npm-shrinkwrap.json doesn’t have a “resolved” field for a package, Shrinkpack could look for the “dist” field in the installed package’s package.json?

I’m actually looking at Shrinkpack’s shrinkwrap.js, and I see that line 58 has if (is.string(object.resolved)), but I don’t actually see it using “resolved” anywhere. A bit confused on that.

[JamieMason]

Totally agree @markerikson, performance just hasn’t been a priority up to now. Initially I just wanted to prove the idea and see if repointing shrinkwrap at relative file paths was even possible, then navigate the edge cases and kinks so it’s all more stable.

With the above now in better shape, performance is now the main priority behind any bugs, should they come up.

Thanks a lot, I’ll create issues from some of your points.

[markerikson]

Just updated my global Shrinkpack to 0.6.0 and running it now. It’s clearly doing something different than it was before. I’m seeing a bunch of new libs being shown as added diffs as it’s running. Also… it’s completely pegging my CPU because it’s firing off all these background NPM tasks. It’s also taking quite a while to run. Now, as I said, my project had at least a couple hundred non-resolved entries in the shrinkwrap file, but it just feels like it oughta be able to chew through this faster. (Admittedly, one issue for me personally is that my company installs a bunch of security and process monitoring software on our machines, and that software insists on inspecting new processes for validity, pegging my CPU as it does so. Running Git commands also pegs my CPU because of the nested process tree.)

Per my earlier comment, would it be possible to have Shrinkpack check /node_modules/some_installed_package/package.json for the “dist” field, which should contain the tarball path? Seems like that might cut down on some of the need to query NPM for the metadata every time. Fewer child processes and network requests.

Still, definitely a step in the right direction as far as “correct” output. Way slower at the moment, but it does look like it should ultimately wind up with all my transitive deps being packed up, whereas before it was clearly missing a large percentage because NPM wasn’t giving enough info.