Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Verdaccio 5.0.3 not working for unix sock behind http1.1 h2 proxy

See original GitHub issue

Reverse proxy setup using unix sock fails with Apache being unable to access service. HAProxy also fails. Apache logs and config should explain it enough. We can look at HAProxy in another issue if this doesn’t resolve it.

In terms of testing, this is a basic browser request to the host which is being proxied.

This setup works perfectly with gitea so should be compatible with verdaccio, it could be a config issue with Verdaccio.

Start command

verdaccio@ninja:~$ export VERDACCIO_PUBLIC_URL=https://npm.example.com/
verdaccio@ninja:~$ cd /var/lib/verdaccio; /usr/bin/node /usr/bin/verdaccio
 warn --- config file  - /var/lib/verdaccio/verdaccio/config.yaml
 warn --- Plugin successfully loaded: verdaccio-mysql
 warn --- Plugin successfully loaded: verdaccio-audit
 warn --- http address - unix:/var/www/npm.example.com/services/verdaccio.sock - verdaccio/5.0.4
(node:652438) Warning: multiple addresses will be deprecated in the next major, only use one
(Use `node --trace-warnings ...` to show where the warning was created)

Debugging output

verdaccio@ninja:~$ exportDEBUG=express:*
verdaccio@ninja:~$ export DEBUG=express:*
verdaccio@ninja:~$ export NODE_DEBUG=request
verdaccio@ninja:~$ cd /var/lib/verdaccio; /usr/bin/node /usr/bin/verdaccio
 warn --- config file  - /var/lib/verdaccio/verdaccio/config.yaml
 warn --- Plugin successfully loaded: verdaccio-mysql
  express:application set "x-powered-by" to true +0ms
  express:application set "etag" to 'weak' +2ms
  express:application set "etag fn" to [Function: generateETag] +0ms
  express:application set "env" to 'development' +1ms
  express:application set "query parser" to 'extended' +0ms
  express:application set "query parser fn" to [Function: parseExtendedQueryString] +0ms
  express:application set "subdomain offset" to 2 +0ms
  express:application set "trust proxy" to false +0ms
  express:application set "trust proxy fn" to [Function: trustNone] +1ms
  express:application booting in development mode +0ms
  express:application set "view" to [Function: View] +0ms
  express:application set "views" to '/var/lib/verdaccio/views' +0ms
  express:application set "jsonp callback name" to 'callback' +0ms
  express:application set "env" to 'production' +0ms
  express:router use '/' query +1ms
  express:router:layer new '/' +0ms
  express:router use '/' expressInit +0ms
  express:router:layer new '/' +0ms
  express:router use '/' corsMiddleware +0ms
  express:router:layer new '/' +0ms
  express:router use '/' <anonymous> +0ms
  express:router:layer new '/' +0ms
  express:router use '/' errorReportingMiddleware +1ms
  express:router:layer new '/' +0ms
  express:router use '/' <anonymous> +0ms
  express:router:layer new '/' +0ms
  express:router use '/' compression +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/static/favicon.ico' +0ms
  express:router:layer new '/-/static/favicon.ico' +0ms
  express:router:route get '/-/static/favicon.ico' +0ms
  express:router:layer new '/' +0ms
 warn --- Plugin successfully loaded: verdaccio-audit
  express:router:route new '/audits' +12ms
  express:router:layer new '/audits' +0ms
  express:router:route post '/audits' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/audits/quick' +1ms
  express:router:layer new '/audits/quick' +0ms
  express:router:route post '/audits/quick' +0ms
  express:router:layer new '/' +0ms
  express:router use '/-/npm/v1/security' router +0ms
  express:router:layer new '/-/npm/v1/security' +0ms
  express:router use '/' <anonymous> +0ms
  express:router:layer new '/' +0ms
  express:router use '/' jsonParser +1ms
  express:router:layer new '/' +0ms
  express:router use '/' <anonymous> +0ms
  express:router:layer new '/' +0ms
  express:router use '/' encodeScopePackage +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/whoami' +0ms
  express:router:layer new '/whoami' +0ms
  express:router:route get '/whoami' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/whoami' +0ms
  express:router:layer new '/-/whoami' +0ms
  express:router:route get '/-/whoami' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/:package/:version?' +1ms
  express:router:layer new '/:package/:version?' +0ms
  express:router:route get '/:package/:version?' +0ms
  express:router:layer new '/' +0ms
  express:router:route get '/:package/:version?' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/:scopedPackage/-/:scope/:filename' +0ms
  express:router:layer new '/:scopedPackage/-/:scope/:filename' +0ms
  express:router:route get '/:scopedPackage/-/:scope/:filename' +0ms
  express:router:layer new '/' +0ms
  express:router:route get '/:scopedPackage/-/:scope/:filename' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/:package/-/:filename' +0ms
  express:router:layer new '/:package/-/:filename' +0ms
  express:router:route get '/:package/-/:filename' +1ms
  express:router:layer new '/' +0ms
  express:router:route get '/:package/-/:filename' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/npm/v1/user' +0ms
  express:router:layer new '/-/npm/v1/user' +0ms
  express:router:route get '/-/npm/v1/user' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/npm/v1/user' +0ms
  express:router:layer new '/-/npm/v1/user' +0ms
  express:router:route post '/-/npm/v1/user' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/all(/since)?' +0ms
  express:router:layer new '/-/all(/since)?' +0ms
  express:router:route get '/-/all(/since)?' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/user/:org_couchdb_user' +0ms
  express:router:layer new '/-/user/:org_couchdb_user' +1ms
  express:router:route get '/-/user/:org_couchdb_user' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/user/:org_couchdb_user/:_rev?/:revision?' +0ms
  express:router:layer new '/-/user/:org_couchdb_user/:_rev?/:revision?' +0ms
  express:router:route put '/-/user/:org_couchdb_user/:_rev?/:revision?' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/user/token/*' +0ms
  express:router:layer new '/-/user/token/*' +0ms
  express:router:route delete '/-/user/token/*' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/_session' +0ms
  express:router:layer new '/_session' +0ms
  express:router:route post '/_session' +0ms
  express:router:layer new '/' +0ms
  express:router:route post '/_session' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/:package/:tag' +1ms
  express:router:layer new '/:package/:tag' +0ms
  express:router:route put '/:package/:tag' +0ms
  express:router:layer new '/' +0ms
  express:router:route put '/:package/:tag' +0ms
  express:router:layer new '/' +0ms
  express:router:route put '/:package/:tag' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/package/:package/dist-tags/:tag' +0ms
  express:router:layer new '/-/package/:package/dist-tags/:tag' +0ms
  express:router:route post '/-/package/:package/dist-tags/:tag' +0ms
  express:router:layer new '/' +0ms
  express:router:route post '/-/package/:package/dist-tags/:tag' +0ms
  express:router:layer new '/' +0ms
  express:router:route post '/-/package/:package/dist-tags/:tag' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/package/:package/dist-tags/:tag' +0ms
  express:router:layer new '/-/package/:package/dist-tags/:tag' +1ms
  express:router:route put '/-/package/:package/dist-tags/:tag' +0ms
  express:router:layer new '/' +0ms
  express:router:route put '/-/package/:package/dist-tags/:tag' +0ms
  express:router:layer new '/' +0ms
  express:router:route put '/-/package/:package/dist-tags/:tag' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/package/:package/dist-tags/:tag' +0ms
  express:router:layer new '/-/package/:package/dist-tags/:tag' +0ms
  express:router:route delete '/-/package/:package/dist-tags/:tag' +0ms
  express:router:layer new '/' +0ms
  express:router:route delete '/-/package/:package/dist-tags/:tag' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/package/:package/dist-tags' +0ms
  express:router:layer new '/-/package/:package/dist-tags' +0ms
  express:router:route get '/-/package/:package/dist-tags' +0ms
  express:router:layer new '/' +0ms
  express:router:route get '/-/package/:package/dist-tags' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/package/:package/dist-tags' +0ms
  express:router:layer new '/-/package/:package/dist-tags' +1ms
  express:router:route post '/-/package/:package/dist-tags' +0ms
  express:router:layer new '/' +0ms
  express:router:route post '/-/package/:package/dist-tags' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/:package/:_rev?/:revision?' +0ms
  express:router:layer new '/:package/:_rev?/:revision?' +0ms
  express:router:route put '/:package/:_rev?/:revision?' +0ms
  express:router:layer new '/' +1ms
  express:router:route put '/:package/:_rev?/:revision?' +0ms
  express:router:layer new '/' +0ms
  express:router:route put '/:package/:_rev?/:revision?' +0ms
  express:router:layer new '/' +0ms
  express:router:route put '/:package/:_rev?/:revision?' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/:package/-rev/*' +0ms
  express:router:layer new '/:package/-rev/*' +0ms
  express:router:route delete '/:package/-rev/*' +0ms
  express:router:layer new '/' +0ms
  express:router:route delete '/:package/-rev/*' +1ms
  express:router:layer new '/' +0ms
  express:router:route new '/:package/-/:filename/-rev/:revision' +0ms
  express:router:layer new '/:package/-/:filename/-rev/:revision' +0ms
  express:router:route delete '/:package/-/:filename/-rev/:revision' +0ms
  express:router:layer new '/' +0ms
  express:router:route delete '/:package/-/:filename/-rev/:revision' +0ms
  express:router:layer new '/' +0ms
  express:router:route delete '/:package/-/:filename/-rev/:revision' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/:package/-/:filename/*' +0ms
  express:router:layer new '/:package/-/:filename/*' +0ms
  express:router:route put '/:package/-/:filename/*' +0ms
  express:router:layer new '/' +0ms
  express:router:route put '/:package/-/:filename/*' +0ms
  express:router:layer new '/' +0ms
  express:router:route put '/:package/-/:filename/*' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/:package/:version/-tag/:tag' +0ms
  express:router:layer new '/:package/:version/-tag/:tag' +0ms
  express:router:route put '/:package/:version/-tag/:tag' +1ms
  express:router:layer new '/' +0ms
  express:router:route put '/:package/:version/-tag/:tag' +0ms
  express:router:layer new '/' +0ms
  express:router:route put '/:package/:version/-tag/:tag' +0ms
  express:router:layer new '/' +0ms
  express:router:route put '/:package/:version/-tag/:tag' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/ping' +0ms
  express:router:layer new '/-/ping' +0ms
  express:router:route get '/-/ping' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/_view/starredByUser' +0ms
  express:router:layer new '/-/_view/starredByUser' +0ms
  express:router:route get '/-/_view/starredByUser' +0ms
  express:router:layer new '/' +0ms
  express:router use '/' router +1ms
  express:router:layer new '/' +0ms
  express:router use '/' <anonymous> +2ms
  express:router:layer new '/' +0ms
  express:router use '/' setSecurityWebHeaders +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/static/*' +0ms
  express:router:layer new '/-/static/*' +0ms
  express:router:route get '/-/static/*' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/-/web/:section/*' +0ms
  express:router:layer new '/-/web/:section/*' +0ms
  express:router:route get '/-/web/:section/*' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/' +0ms
  express:router:layer new '/' +0ms
  express:router:route get '/' +0ms
  express:router:layer new '/' +0ms
  express:router use '/' router +0ms
  express:router:layer new '/' +0ms
  express:router use '/' urlencodedParser +1ms
  express:router:layer new '/' +0ms
  express:router use '/' <anonymous> +0ms
  express:router:layer new '/' +0ms
  express:router use '/' setSecurityWebHeaders +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/packages' +0ms
  express:router:layer new '/packages' +0ms
  express:router:route get '/packages' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/package/readme/(@:scope/)?:package/:version?' +0ms
  express:router:layer new '/package/readme/(@:scope/)?:package/:version?' +1ms
  express:router:route get '/package/readme/(@:scope/)?:package/:version?' +0ms
  express:router:layer new '/' +0ms
  express:router:route get '/package/readme/(@:scope/)?:package/:version?' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/sidebar/(@:scope/)?:package' +0ms
  express:router:layer new '/sidebar/(@:scope/)?:package' +0ms
  express:router:route get '/sidebar/(@:scope/)?:package' +0ms
  express:router:layer new '/' +0ms
  express:router:route get '/sidebar/(@:scope/)?:package' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/search/:anything' +0ms
  express:router:layer new '/search/:anything' +0ms
  express:router:route get '/search/:anything' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/login' +0ms
  express:router:layer new '/login' +0ms
  express:router:route post '/login' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/reset_password' +0ms
  express:router:layer new '/reset_password' +0ms
  express:router:route put '/reset_password' +0ms
  express:router:layer new '/' +0ms
  express:router use '/-/verdaccio/' router +1ms
  express:router:layer new '/-/verdaccio/' +0ms
  express:router:route new '/*' +0ms
  express:router:layer new '/*' +0ms
  express:router:route get '/*' +0ms
  express:router:layer new '/' +0ms
  express:router use '/' <anonymous> +0ms
  express:router:layer new '/' +0ms
  express:router use '/' final +0ms
  express:router:layer new '/' +0ms
 warn --- http address - unix:/var/www/npm.example.com/services/verdaccio.sock - verdaccio/5.0.4
(node:655795) Warning: multiple addresses will be deprecated in the next major, only use one
(Use `node --trace-warnings ...` to show where the warning was created)

Apache Logs

+651630:608d54de:1|GET / HTTP/1.1|Host:npm.example.com|Connection:keep-alive|Cache-Control:max-age=0|Upgrade-Insecure-Requests:1|User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85
Safari/537.36|Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9|Accept-Encoding:gzip, deflate|Accept-Language:en,en-GB;q=0.9

==> /var/www/npm.example.com/logs/apache2.error.log <==
[Sat May 01 13:17:18.982418 2021] [proxy_fcgi:error] [pid 651630:tid 140649075480320] [client 192.168.88.2:65300] AH01068: Got bogus version 72

[Sat May 01 13:17:18.982470 2021] [proxy_fcgi:error] [pid 651630:tid 140649075480320] (22)Invalid argument: [client 192.168.88.2:65300] AH01075: Error dispatching request to :

==> /var/www/npm.example.com/logs/apache2.access.log <==
192.168.88.2 - - [01/May/2021:13:17:18 +0000] "GET / HTTP/1.1" 503 573 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"

==> /var/www/npm.example.com/logs/apache2.forensic.log <==
-651630:608d54de:1

Apache config

<VirtualHost *:80>
  ServerName npm.example.com
  ServerAdmin admin@example.com
  DocumentRoot /var/www/npm.example.com/htdocs
  Protocols http/1.1 h2
  ForensicLog /var/www/npm.example.com/logs/apache2.forensic.log
  LogLevel warn
  ErrorLog /var/www/npm.example.com/logs/apache2.error.log
  CustomLog /var/www/npm.example.com/logs/apache2.access.log combined

  KeepAliveTimeout 1
  ProxyPass "/" "unix:///var/www/npm.example.com/services/verdaccio.sock|fcgi://npm.example.com:4853/" nocanon
  ProxyPassReverse "/" "http://npm.example.com/"
  ProxyPreserveHost On
  ProxyRequests off
  AllowEncodedSlashes NoDecode

  SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
  SetEnv proxy-chain-auth On
</VirtualHost>

Configuration File (cat ~/.config/verdaccio/config.yaml)

verdaccio@ninja:~$ cat /var/lib/verdaccio/verdaccio/config.yaml


storage: ./storage
plugins: ./plugins

url_prefix: /
listen:
  - unix:/var/www/npm.example.com/services/verdaccio.sock

http_proxy: http://npm.example.com/
https_proxy: https://npm.example.com/

web:
  title: Flipkick Node.js Repository
  darkMode: true
  favicon: https://example.com/favicon.ico

auth:
    mysql:
        connection:
            host: "127.0.0.1"
            port: 3306
            user: "verdaccio"
            password: “password”
            database: "verdaccio"
        queries:
            auth_user: 'SELECT QUERY'
            add_user: 'INSERT QUERY'
            update_user: 'UPDATE QUERY'

uplinks:
  npmjs:
    url: https://registry.npmjs.org/

packages:
  '@*/*':
    # scoped packages
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs

  '**':
    access: $all
    publish: $authenticated
    unpublish: $authenticated
    proxy: npmjs
server:
  keepAliveTimeout: 60

middlewares:
  audit:
    enabled: true

logs: { type: stdout, format: pretty, level: http }

verdaccio@ninja:~$

Environment information

verdaccio@ninja:~$ verdaccio --info

Environment Info:
  System:
    OS: Linux 5.4 Ubuntu 20.04.2 LTS (Focal Fossa)
    CPU: (4) x64 Intel(R) Core(TM) i5-3570S CPU @ 3.10GHz
  Binaries:
    npm: 7.11.2 - /usr/bin/npm
  Virtualization:
    Docker: 20.10.6 - /usr/bin/docker
  npmGlobalPackages:
    verdaccio: 5.0.4

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:10 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
ghostcommented, May 3, 2021

This was a configuration error on my proxy. The proxying provided by Verdaccio is not fastcgi based, its plain old HTTP based. Could fastcgi be implemented? Its certainly a much better way to provide a proxied service and is significantly faster as the service doesn’t need to handle HTTP processing and allows the proxy to bear some of that load.

0reactions
ghostcommented, May 3, 2021

A socat debug session between the proxy and the server shows the following:

socat -v UNIX-LISTEN:./socat-verdaccio.sock UNIX-CONNECT:/var/www/npm.example.com/services/verdaccio.sock

> 2021/05/03 09:49:56.787518  length=1227 from=0 to=1226
.....\b....................HTTP_AUTHORIZATION..proxy-chain-authOn	.HTTP_HOSTnpm.example.com.
HTTP_CONNECTIONkeep-alive.	HTTP_CACHE_CONTROLmax-age=0..HTTP_UPGRADE_INSECURE_REQUESTS1.xHTTP_USER_AGENTMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36\v....HTTP_ACCEPTtext/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9.\rHTTP_ACCEPT_ENCODINGgzip, deflate..HTTP_ACCEPT_LANGUAGEen,en-GB;q=0.9.FPATH/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin.OSERVER_SIGNATURE<address>Apache/2.4.41 (Ubuntu) Server at npm.example.com Port 80</address>
..SERVER_SOFTWAREApache/2.4.41 (Ubuntu)\v.SERVER_NAMEnpm.example.com\v.SERVER_ADDR192.168.88.254\v.SERVER_PORT80\v\fREMOTE_ADDR192.168.88.2\r"DOCUMENT_ROOT/var/www/npm.example.com/htdocs..REQUEST_SCHEMEhttp..CONTEXT_PREFIX."CONTEXT_DOCUMENT_ROOT/var/www/npm.example.com/htdocs\f.SERVER_ADMINadmin@example.com.%SCRIPT_FILENAMEproxy:fcgi://npm.example.com:4853/\v.REMOTE_PORT58086.\aGATEWAY_INTERFACECGI/1.1.\bSERVER_PROTOCOLHTTP/1.1..REQUEST_METHODGET\f.QUERY_STRING\v.REQUEST_URI/\v.SCRIPT_NAME/................< 2021/05/03 09:49:56.790887  length=47 from=0 to=46
HTTP/1.1 400 Bad Request\r
Connection: close\r
\r
Read more comments on GitHub >

github_iconTop Results From Across the Web

[V5] Verdaccio is not working behind http2 proxy #2190 - GitHub
Describe the bug Verdaccio v5 is not detecting headers passed from reverse proxy, I think it's because of http2 is lowercasing all headers ......
Read more >
Reverse Proxy Setup - Verdaccio
A possible fix for this, can be by disabling gzip compression for the virtual host, by adding this to your config: SetEnv no-gzip...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Verdaccio not working as expected after path mapping in a GCE Ingress controller

Next page

Configure Pino.js to work with pm2