htpasswd plugin bcrypt synchronous blocking API call
See original GitHub issueYour Environment
- verdaccio version: 5.x.x, 6.xxx
- node version: 16.x
- package manager: N/A
- os: N/A
- platform: N/A
Describe the bug
This bug report is a follow-up issue to a previous discussion in https://github.com/verdaccio/charts/issues/90 related to bcrypt and password verification (specifically this comment). The default htpasswd
plugin used by both Verdaccio 5.x and 6.x makes use of the bcryptjs
compareSync
API. This can be seen here:
- Verdaccio 6: https://github.com/verdaccio/verdaccio/blob/d2f9013ece8fc3b37e16e9748d03bce2acd7aba5/packages/plugins/htpasswd/src/utils.ts#L59
- Verdaccio 5: verdaccio/monorepo/plugins/htpasswd/src/utils.ts#L46
Usage of the compareSync
API call instead of the async compare
API call generally isn’t a problem except for certain circumstances that we recently ran into. I will explain more below.
The htpasswd
authentication plugin uses 10
salt rounds by default when creating passwords (see here). This is a fairly reasonable/typical value when generating bcrypt hashed passwords.
Our Verdaccio instance has user registration disabled and we generate the users and associated bcrypt hashed passwords using a different tool. We were seeing what we thought were random CPU spikes. When these CPU spikes occurred Verdaccio would stop responding to most HTTP requests as well. After much troubleshooting and debugging we found that:
- Verdaccio was spending most of it’s CPU cycles and time on the
compareSync
API call to verify the user password on authenticated requests. After checking, we realized that our passwords were being generated using 15 salt rounds, which is relatively high. - When Verdaccio was processing a large number of concurrent authenticated requests with passwords that had been generated using a higher then normal number of salt rounds, the following behavior was observed:
- Verdaccio would begin responding to requests in an almost sequential manner (FIFO)
- Verdaccio would not respond to any new incoming requests. This included health check requests made by our k8s cluster.
It makes sense that CPU usage would increase when verifying user passwords generated with a higher number of salt rounds. The cost to verify the password is directly related to the number of salt rounds used to hash the password. However, the issues described in bullet point <span>#</span>2 above could be mostly avoided/mitigated by using the non-blocking bcryptjs
compareAsync
API call instead.
I did some bench-marking where 36 requests were sent to Verdaccio concurrently. Each request was authenticated using a username/password (as opposed to a JWT token). The test was run on an AMD Ryzen 7 CPU where each core was 3.0GHZ (core count doesn’t matter since NodeJS is single threaded). Below were the results:
Password hash salt rounds | Seconds to complete 36 requests |
---|---|
15 | 75 |
10 | 3 |
8 | 1 |
Based on these results I would argue that the default value of 10
salt rounds is still too high as 3 seconds to process 36 requests is not great.
I think there are a few changes that can be made which would mitigate the issues described above and prevent other users from encountering the same problem:
- Switch the
bcryptjs
compareSync
API call tocompare
in order to not block the NodeJS event loop and allow Verdaccio to respond to other requests when verifying authenticated username/password requests (especially in scenarios where the password may have been generated with a high number of salt rounds) - Reduce the default
htpasswd.rounds
configuration value from 10 to 8. - Provide some information in the README.md file for the
htpasswd
plugin as to the effects of increasing therounds
configuration value. - Add some logic to the
htpasswd
plugin to capture the duration of time it takes to verify a user password andlog.warn
if the time it takes is over a certain threshold. This would indicate that passwords are being generated using too many salt rounds.
To Reproduce See description above.
Screenshots, server logs, package manager log N/A
Configuration File (cat ~/.config/verdaccio/config.yaml) N/A
Environment information N/A
Debugging output N/A
- I’m willing to fix this bug 🥇
Issue Analytics
- State:
- Created 2 years ago
- Comments:10 (5 by maintainers)
Top GitHub Comments
That would be fantastic, especially from the perspective of debugging and getting a profiler attached when necessary.
Yeah, I think it’s safe to close. It’s on my TODO list to get this deployed but I’ve had zero time this past week. Thanks.