Package locking is crazy slow for scikit-learn

I’m sorry I don’t have a reduced test case for this but this is so crazy slow it’s hard to actually debug.

Steps to reproduce

1. Clone https://github.com/HearthSim/hearthsim-vagrant
2. Inside that directory, clone https://github.com/HearthSim/HSReplay.net
3. Run docker-compose build django (this builds an image based on the python-stretch docker image, which will also install the latest pipenv systemwide, cf. Dockerfile).
4. Finally, run docker-compose run django, which runs pipenv install --dev

On linux, this stays stuck at Locking [packages] for over 15 minutes, with no output even when run with --verbose. Then after ~15 mins, it gives me the full output of what it’s been doing for all that time. When run outside of docker, it still takes a couple of minutes on that step, but at most 1-2 mins. I have a pretty beefy CPU and SSD, so I don’t know why it would take this long in the first place.

I also see a lot of Warning: Error generating hash for ... in the verbose output, I don’t know if that’s related.

Any idea? How can I debug this further?

[kennethreitz]

you shouldn’t be running lock in docker in the first place…

[jleclanche]

Cheers @jleclanche I used to play hearthstone in its early days before there was much tooling around it so thanks for building some community tools!

😃

That’s the whole purpose here— you already have an isolated python environment in a virtualenv which is managed by pipenv, so it can handle dependency graphs across platforms, os and whatnot. S

I fully understand what pipenv brings to the table. Just to explain why I’m using it in docker:

• I need docker because the stack I’m running locally is complex. It’s not just a single python app, it’s a python app, database, mock servers, redis server, etc. All these need to be available, cross-platform, consistently between all devs on the team. Docker solves that.
• I need (want) pipenv because I need (want) to track my dependencies in a pipfile, rather than requirements.txt. That is to say, I’m moving the app to pipenv anyway. So now my choice is to either duplicate the dependencies, or use pipenv consistently in docker as well.

With that said, I’m not interested in solving my problem. I solved my problem by adding --skip-lock. I’m interested in solving, or helping solve, the egregious difference in performance between inside and outside of the container. Or at least coming out of this with a “there’s a very good reason for this difference and here it is”.

But yarn is also running inside that same container and managing 1-2 orders of magnitude more dependencies than pipenv, so I think we can do better. And if that takes me PRing setup.py/setup.cfg fixes to 30 different projects so be it 😃