NPM7: Cannot read property pickAlgorithm of null / tarball seems to be corrupted

Your Environment

• verdaccio version: 5.1.2
• node version: 16.5.0
• package manager: npm@7.19.1 - npm@7.20.6 (haven’t tested earlier 7.x versions, however npm@6 does work)
• os: [mac, windows@10, linux]: Linux Ubuntu 20.04
• platform: [npm, docker, helm, other] npm

Describe the bug

NPM fails to install packages when the registry is set to the verdaccio server (v5.1.2) when using NPM 7. NPM 6 does work / node 12 does work against the same server. Attempting to upgrade my infrastructure to use Node 16 (which by default comes with NPM 7) produces a failure.

In my existing projects, every package attempted to downloaded/install fails with a line that looks similar to:

npm WARN tarball tarball data for debug@http://<redacted-verdaccio-server-host>/debug/-/debug-2.6.9.tgz (sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==) seems to be corrupted. Trying again.

After some time, NPM eventually fails with the error message:

verb stack TypeError: Cannot read property 'pickAlgorithm' of null
npm verb stack     at Integrity.match (/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/lib/node_modules/npm/node_modules/ssri/index.js:234:24)
npm verb stack     at CachePolicy.satisfies (/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/lib/node_modules/npm/node_modules/make-fetch-happen/lib/cache/policy.js:113:49)
npm verb stack     at Function.find (/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/lib/node_modules/npm/node_modules/make-fetch-happen/lib/cache/entry.js:172:25)
npm verb stack     at async cacheFetch (/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/lib/node_modules/npm/node_modules/make-fetch-happen/lib/cache/index.js:8:17)
npm verb stack     at async fetch (/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/lib/node_modules/npm/node_modules/make-fetch-happen/lib/fetch.js:82:7)
npm verb cwd /home/<redacted>/test
npm verb Linux 4.15.0-153-generic
npm verb argv "/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/bin/node" "/opt/node/runtime/bin/npm" "install" "tslib" "--verbose"
npm verb node v16.5.0
npm verb npm  v7.20.6
npm ERR! Cannot read property 'pickAlgorithm' of null
npm verb exit 1

To simplify the situation, I created an empty folder and tried to just simply install single package, which produces the Cannot read property 'pickAlgorithm' of null like above.

NPM does work however, if I set the registry back to https://registry.npmjs.org by running:

npm --registry https://registry.npmjs.org install tslib --verbose

For this reason, I believe something changed in NPM 7 in a way that the client expects the server to do something.

To Reproduce With npm@7 installed:

mkdir test
cd test
npm install typescript

Should be the simpliest way to trigger the error.

Expected behavior

For npm to install packages correctly through verdaccio

Screenshots, server logs, package manager log

NPM 7 verbose output (error)

npm install tslib --verbose
npm verb cli [
npm verb cli   '/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/bin/node',
npm verb cli   '/opt/node/runtime/bin/npm',
npm verb cli   'install',
npm verb cli   'tslib',
npm verb cli   '--verbose'
npm verb cli ]
npm info using npm@7.20.6
npm info using node@v16.5.0
npm timing npm:load:whichnode Completed in 0ms
npm timing config:load:defaults Completed in 1ms
npm timing config:load:file:/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/lib/node_modules/npm/npmrc Completed in 1ms
npm timing config:load:builtin Completed in 1ms
npm timing config:load:cli Completed in 2ms
npm timing config:load:env Completed in 0ms
npm timing config:load:file:/home/<redacted>/test/.npmrc Completed in 1ms
npm timing config:load:project Completed in 1ms
npm timing config:load:file:/home/<redacted>/.npmrc Completed in 1ms
npm timing config:load:user Completed in 1ms
npm timing config:load:file:/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/etc/npmrc Completed in 0ms
npm timing config:load:global Completed in 0ms
npm timing config:load:validate Completed in 1ms
npm timing config:load:credentials Completed in 1ms
npm timing config:load:setEnvs Completed in 1ms
npm timing config:load Completed in 9ms
npm timing npm:load:configload Completed in 9ms
npm timing npm:load:setTitle Completed in 0ms
npm timing npm:load:setupLog Completed in 0ms
npm timing config:load:flatten Completed in 2ms
npm timing npm:load:cleanupLog Completed in 2ms
npm timing npm:load:configScope Completed in 0ms
npm timing npm:load:projectScope Completed in 0ms
npm timing npm:load Completed in 17ms
npm timing arborist:ctor Completed in 0ms
npm timing idealTree:init Completed in 10ms
npm timing idealTree:userRequests Completed in 1ms
npm verb shrinkwrap failed to load node_modules/.package-lock.json out of date, updated: node_modules
npm http fetch GET 200 https://<redacted-verdaccio-host>:48731/tslib 300ms (cache updated)
npm timing idealTree:#root Completed in 308ms
npm timing idealTree:node_modules/tslib Completed in 0ms
npm timing idealTree:buildDeps Completed in 309ms
npm timing idealTree:fixDepFlags Completed in 0ms
npm timing idealTree Completed in 322ms
npm timing reify:loadTrees Completed in 322ms
npm timing reify:diffTrees Completed in 2ms
npm timing reify:retireShallow Completed in 0ms
npm timing reify:createSparse Completed in 0ms
npm timing reify:loadBundles Completed in 0ms
npm timing reify:rollback:createSparse Completed in 0ms
npm timing reify:rollback:retireShallow Completed in 0ms
npm timing command:install Completed in 363ms
npm verb stack TypeError: Cannot read property 'pickAlgorithm' of null
npm verb stack     at Integrity.match (/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/lib/node_modules/npm/node_modules/ssri/index.js:234:24)
npm verb stack     at CachePolicy.satisfies (/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/lib/node_modules/npm/node_modules/make-fetch-happen/lib/cache/policy.js:113:49)
npm verb stack     at Function.find (/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/lib/node_modules/npm/node_modules/make-fetch-happen/lib/cache/entry.js:172:25)
npm verb stack     at async cacheFetch (/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/lib/node_modules/npm/node_modules/make-fetch-happen/lib/cache/index.js:8:17)
npm verb stack     at async fetch (/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/lib/node_modules/npm/node_modules/make-fetch-happen/lib/fetch.js:82:7)
npm verb cwd /home/<redacted>/test
npm verb Linux 4.15.0-153-generic
npm verb argv "/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/bin/node" "/opt/node/runtime/bin/npm" "install" "tslib" "--verbose"
npm verb node v16.5.0
npm verb npm  v7.20.6
npm ERR! Cannot read property 'pickAlgorithm' of null
npm verb exit 1
npm timing npm Completed in 525ms
npm verb unfinished npm timer reify 1629227192871
npm verb unfinished npm timer reify:audit 1629227193214
npm verb unfinished npm timer auditReport:getReport 1629227193214
npm verb unfinished npm timer reify:unpack 1629227193215
npm verb unfinished npm timer reifyNode:node_modules/tslib 1629227193216
npm verb code 1

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/<redacted>/.npm/_logs/2021-08-17T19_06_33_235Z-debug.log

NPM 7 install with registry set to NPM official registry

npm --registry https://registry.npmjs.org install tslib --verbose
npm verb cli [
npm verb cli   '/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/bin/node',
npm verb cli   '/opt/node/runtime/bin/npm',
npm verb cli   '--registry',
npm verb cli   'https://registry.npmjs.org',
npm verb cli   'install',
npm verb cli   'tslib',
npm verb cli   '--verbose'
npm verb cli ]
npm info using npm@7.20.6
npm info using node@v16.5.0
npm timing npm:load:whichnode Completed in 0ms
npm timing config:load:defaults Completed in 1ms
npm timing config:load:file:/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/lib/node_modules/npm/npmrc Completed in 0ms
npm timing config:load:builtin Completed in 1ms
npm timing config:load:cli Completed in 3ms
npm timing config:load:env Completed in 0ms
npm timing config:load:file:/home/<redacted>/test/.npmrc Completed in 0ms
npm timing config:load:project Completed in 1ms
npm timing config:load:file:/home/<redacted>/.npmrc Completed in 1ms
npm timing config:load:user Completed in 1ms
npm timing config:load:file:/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/etc/npmrc Completed in 1ms
npm timing config:load:global Completed in 1ms
npm timing config:load:validate Completed in 0ms
npm timing config:load:credentials Completed in 1ms
npm timing config:load:setEnvs Completed in 0ms
npm timing config:load Completed in 9ms
npm timing npm:load:configload Completed in 9ms
npm timing npm:load:setTitle Completed in 1ms
npm timing npm:load:setupLog Completed in 0ms
npm timing config:load:flatten Completed in 2ms
npm timing npm:load:cleanupLog Completed in 2ms
npm timing npm:load:configScope Completed in 0ms
npm timing npm:load:projectScope Completed in 0ms
npm timing npm:load Completed in 18ms
npm timing arborist:ctor Completed in 1ms
npm timing idealTree:init Completed in 9ms
npm timing idealTree:userRequests Completed in 1ms
npm verb shrinkwrap failed to load node_modules/.package-lock.json out of date, updated: node_modules
npm http fetch GET 200 https://registry.npmjs.org/tslib 199ms (cache miss)
npm timing idealTree:#root Completed in 204ms
npm timing idealTree:node_modules/tslib Completed in 0ms
npm timing idealTree:buildDeps Completed in 205ms
npm timing idealTree:fixDepFlags Completed in 0ms
npm timing idealTree Completed in 218ms
npm timing reify:loadTrees Completed in 218ms
npm timing reify:diffTrees Completed in 1ms
npm timing reify:retireShallow Completed in 0ms
npm timing reify:createSparse Completed in 1ms
npm timing reify:loadBundles Completed in 0ms
npm http fetch POST 200 https://registry.npmjs.org/-/npm/v1/security/advisories/bulk 202ms
npm timing auditReport:getReport Completed in 202ms
npm timing auditReport:init Completed in 0ms
npm timing reify:audit Completed in 204ms
npm http fetch GET 200 https://registry.npmjs.org/tslib/-/tslib-2.3.1.tgz 206ms (cache miss)
npm timing reifyNode:node_modules/tslib Completed in 212ms
npm timing reify:unpack Completed in 213ms
npm timing reify:unretire Completed in 0ms
npm timing build:queue Completed in 0ms
npm timing build:deps Completed in 1ms
npm timing build Completed in 1ms
npm timing reify:build Completed in 1ms
npm timing reify:trash Completed in 0ms
npm timing reify:save Completed in 4ms
npm timing reify Completed in 463ms

added 1 package, and audited 3 packages in 622ms

found 0 vulnerabilities
npm timing command:install Completed in 466ms
npm verb exit 0
npm timing npm Completed in 623ms
npm info ok

Configuration File (cat ~/.config/verdaccio/config.yaml)

My YAML config

# Path to the directory with all packages
storage: ./storage

# Path to the directory with plugins to include
plugins: ./plugins

listen:
  - 127.0.0.1:48730

max_body_size: 1024mb

web:
  enable: false

auth:
  htpasswd:
    file: ./htpasswd
    max_users: -1 # -1 disables registration

uplinks:
  npmjs:
    url: https://registry.npmjs.org

packages:
  '@<redacted>/**':
    access: <redacted>
    publish: <redacted>
    unpublish: <redacted>

  '@*/*':
    access: <redacted>
    publish: <redacted>
    unpublish: <redacted>
    proxy: npmjs

  '**':
    access: <redacted>
    publish: <redacted>
    unpublish: <redacted>
    proxy: npmjs

server:
  keepAliveTimeout: 1024

middlewares:
  audit:
    enabled: true

logs:
  - { type: stdout, format: pretty, level: http }
  - { type: file, path: /var/log/verdaccio/verdaccio.log, level: info }

Environment information

Environment Info:
  System:
    OS: Linux 4.15 Ubuntu 18.04.5 LTS (Bionic Beaver)
    CPU: (2) x64 Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
  Binaries:
    npm: 6.14.13 - /opt/node/runtime/bin/npm

npm: 6.14.13 - /opt/node/runtime/bin/npm

Would the verdaccio server only having NPM 6.x installed be the cause of the problem??

Debugging output

• $ NODE_DEBUG=request verdaccio display request calls (verdaccio <–> uplinks)

NODE_DEBUG=request verdaccio - npm install typescript

``` http --- 127.0.0.1 requested 'GET /typescript' info --- auth/allow_action: access granted to: undefined info --- allowed access for typescript http --- 304, user: <redacted>(47.55.225.164 via 127.0.0.1), req: 'GET /typescript', bytes: 0/0 http --- 127.0.0.1 requested 'POST /-/npm/v1/security/advisories/bulk' http --- 404, user: <redacted>(47.55.225.164 via 127.0.0.1), req: 'POST /-/npm/v1/security/advisories/bulk', bytes: 44/173 ```

• $ DEBUG=verdaccio* verdaccio enable extreme verdaccio debug mode (verdaccio api)

verdaccio extremel debug mode output

``` verdaccio:logger setup logger +0ms verdaccio:logger logging stdout enabled +1ms verdaccio:logger has prettifier? true +1ms warn --- config file - /var/lib/verdaccio/.config/verdaccio/config.yaml verdaccio:logger setup logger +929ms verdaccio:logger logging stdout enabled +0ms verdaccio:logger has prettifier? true +1ms verdaccio:plugin:local-storage sync database started +0ms verdaccio:plugin:local-storage sync folder '/var/lib/verdaccio/storage' created succeed +1ms verdaccio:plugin:local-storage sync write succeed +0ms (node:10493) Warning: deprecate: multiple logger configuration is deprecated, please check the migration guide. verdaccio:plugin:local-storage sync database started +4ms verdaccio:plugin:local-storage sync folder '/var/lib/verdaccio/storage' created succeed +1ms verdaccio:plugin:local-storage sync write succeed +0ms warn --- Plugin successfully loaded: verdaccio-htpasswd warn --- Plugin successfully loaded: verdaccio-audit warn --- http address - http://127.0.0.1:48730/ - verdaccio/5.1.2 (node:10493) Warning: multiple addresses will be deprecated in the next major, only use one http --- 127.0.0.1 requested 'GET /typescript' verdaccio:auth api middleware using legacy auth token +0ms verdaccio:auth authenticating '<redacted>' +1ms (node:10493) [DEP0106] DeprecationWarning: crypto.createDecipher is deprecated. verdaccio:auth authentication for user '<redacted>' was successfully. Groups: [ '<redacted>' ] +14ms verdaccio [middleware/allow]['access'] allow for '<redacted>' +0ms verdaccio:auth allow access for 'typescript' +5ms verdaccio [auth/allow_action]: user: '<redacted>' +0ms verdaccio [auth/allow_action]: hasPermission? true} for user: '<redacted>' +0ms info --- auth/allow_action: access granted to: undefined info --- allowed access for typescript verdaccio:plugin:local-storage storage path selected: ./storage +10s verdaccio:plugin:local-storage storage absolute path: /var/lib/verdaccio/.config/verdaccio/storage/typescript +0ms verdaccio:plugin:local-storage:fs read a package 'typescript' +0ms verdaccio:plugin:local-storage:fs reading the file: '/var/lib/verdaccio/.config/verdaccio/storage/typescript/package.json' +1ms verdaccio:plugin:local-storage:fs read file '/var/lib/verdaccio/.config/verdaccio/storage/typescript/package.json' succeed +7ms verdaccio:plugin:local-storage:fs read storage file 'typescript' has succeed +102ms info --- making request: 'GET https://registry.npmjs.org/typescript' http --- 304, req: 'GET https://registry.npmjs.org/typescript' (streaming) http --- 304, req: 'GET https://registry.npmjs.org/typescript', bytes: 0/0 verdaccio:plugin:local-storage storage path selected: ./storage +187ms verdaccio:plugin:local-storage storage absolute path: /var/lib/verdaccio/.config/verdaccio/storage/typescript +1ms verdaccio:plugin:local-storage:fs read a package 'typescript' +78ms verdaccio:plugin:local-storage:fs reading the file: '/var/lib/verdaccio/.config/verdaccio/storage/typescript/package.json' +0ms verdaccio:plugin:local-storage:fs read file '/var/lib/verdaccio/.config/verdaccio/storage/typescript/package.json' succeed +7ms verdaccio:plugin:local-storage:fs read storage file 'typescript' has succeed +58ms verdaccio:local-storage updating package 'typescript' info +0ms verdaccio:plugin:local-storage storage path selected: ./storage +68ms verdaccio:plugin:local-storage storage absolute path: /var/lib/verdaccio/.config/verdaccio/storage/typescript +0ms verdaccio:plugin:local-storage:fs save a package 'typescript' +3ms verdaccio:plugin:local-storage:fs creating a new file:: '/var/lib/verdaccio/.config/verdaccio/storage/typescript/package.json' +91ms verdaccio req protocol: undefined +0ms verdaccio combined protocol 'http' +4ms verdaccio combined host '<redacted>' +0ms verdaccio combined prefix '' +0ms verdaccio combined url 'http://<redacted>/' +0ms verdaccio public url by request 'http://<redacted>/' +0ms http --- 200, user: <redacted>(47.55.225.164 via 127.0.0.1), req: 'GET /typescript', bytes: 0/1018999 http --- 127.0.0.1 requested 'POST /-/npm/v1/security/advisories/bulk' verdaccio:auth api middleware using legacy auth token +1s verdaccio:auth authenticating '<redacted>' +0ms verdaccio:auth authentication for user '<redacted>' was successfully. Groups: [ '<redacted>' ] +13ms http --- 404, user: <redacted>(47.55.225.164 via 127.0.0.1), req: 'POST /-/npm/v1/security/advisories/bulk', bytes: 44/173 ```

• $ npm -ddd prints:

npm -ddd output

npm -ddd
npm verb cli [
npm verb cli   '/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/bin/node',
npm verb cli   '/opt/node/runtime/bin/npm',
npm verb cli   '-ddd'
npm verb cli ]
npm info using npm@7.20.6
npm info using node@v16.5.0
npm timing npm:load:whichnode Completed in 0ms
npm timing config:load:defaults Completed in 1ms
npm timing config:load:file:/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/lib/node_modules/npm/npmrc Completed in 1ms
npm timing config:load:builtin Completed in 1ms
npm timing config:load:cli Completed in 2ms
npm timing config:load:env Completed in 0ms
npm timing config:load:file:/home/<redacted>/test/.npmrc Completed in 0ms
npm timing config:load:project Completed in 4ms
npm timing config:load:file:/home/<redacted>/.npmrc Completed in 1ms
npm timing config:load:user Completed in 1ms
npm timing config:load:file:/opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/etc/npmrc Completed in 1ms
npm timing config:load:global Completed in 1ms
npm timing config:load:validate Completed in 1ms
npm timing config:load:credentials Completed in 1ms
npm timing config:load:setEnvs Completed in 0ms
npm timing config:load Completed in 12ms
npm timing npm:load:configload Completed in 12ms
npm timing npm:load:setTitle Completed in 0ms
npm timing npm:load:setupLog Completed in 1ms
npm timing config:load:flatten Completed in 2ms
npm timing npm:load:cleanupLog Completed in 2ms
npm timing npm:load:configScope Completed in 0ms
npm timing npm:load:projectScope Completed in 0ms
npm timing npm:load Completed in 22ms
npm <command>

Usage:

npm install        install all the dependencies in your project
npm install <foo>  add the <foo> dependency to your project
npm test           run this project's tests
npm run <foo>      run the script named <foo>
npm <command> -h   quick help on <command>
npm -l             display usage info for all commands
npm help <term>    search for help on <term>
npm help npm       more involved overview

All commands:

    access, adduser, audit, bin, bugs, cache, ci, completion,
    config, dedupe, deprecate, diff, dist-tag, docs, doctor,
    edit, exec, explain, explore, find-dupes, fund, get, help,
    hook, init, install, install-ci-test, install-test, link,
    ll, login, logout, ls, org, outdated, owner, pack, ping,
    pkg, prefix, profile, prune, publish, rebuild, repo,
    restart, root, run-script, search, set, set-script,
    shrinkwrap, star, stars, start, stop, team, test, token,
    uninstall, unpublish, unstar, update, version, view, whoami

Specify configs in the ini-formatted file:
    /home/<redacted>/.npmrc
or on the command line via: npm <command> --key=value

More configuration info: npm help config
Configuration fields: npm help 7 config

npm@7.20.6 /opt/node/node-v16.5.0-linux-x64/node-v16.5.0-linux-x64/lib/node_modules/npm
npm verb exit 0
npm timing npm Completed in 118ms
npm verb code 1

• $ npm config get registry prints: https://<redacted-domain>:48731/

Port 48731 is an NGINX server, which proxies request to port 48730, which is the verdaccio node server.

Contribute to Verdaccio

• I’m willing to fix this bug 🥇 - I’m willing to help in anyway I can, but I’m not familiar with NPM / verdaccio’s codebase whatsoever.

Additional Information

In my time of writing this bug report, I realized that if you install a package while overriding the registry using npm’s official registry, then retry installing the package using the verdaccio server, it will work, presumably because of some cache.

In fact, running npm cache clean --force and attempting to reinstall the package on NPM 7 against a verdaccio server will make this issue re-appear.

I’ve also seen some older bug reports but they all appear to occur on NPM 6, so I don’t believe this is the same issue, especially since Verdaccio does work for me when using NPM 6 / Node 12.

[breautek]

Yup… so I was comparing my nginx configs with the `verdaccio docs and noticed I had some discrepencies…

I was completely missing proxy_set_header X-Forwarded-Proto $scheme; so I added that… but didn’t solve the problem.

Then I found the aforementioned config about running the server under a different domain/port (in my case… I have a unique set port since I think the default port conflicts with other services on the box).

I had proxy_set_header Host $host;

instead of…

proxy_set_header Host $host:$server_port;

Like suggested in the Verdaccio’s docs. After correcting that, that appeared to fix everything. Had no idea how or why it worked under NPM 6 🤷 but actually following the docs seems to work 👍

[breautek]

Yuuup… the NGINX proxy definitely has something to do with this I guess. If I open up traffic directly to the node verdaccio server and configure NPM client to connect directly to the verdaccio server, things do work as expected.

I’ll close this issue cause I think it’s obvious now that it’s not actually a verdaccio issue. Sorry, should have conducted this test earlier 😅 – if I find the cause, I’ll post it here just in case anybody else have a similar issue.

I suspect maybe a certain URL not being proxied properly.

Thanks for taking a look though. You responded awfully quick 😃