Decreased performance after replacing multiprocessing.dummy with concurrent.futures

I did a quick test with my code from https://github.com/networklore/networklore-demos/tree/master/python/nornir/ansible-nornir-speed

pip install nornir==2.3.0
time python nornir-run.py 5000
(ran three times)
python nornir-run.py 5000  10.49s user 3.46s system 116% cpu 12.006 total
python nornir-run.py 5000  10.65s user 3.27s system 119% cpu 11.602 total
python nornir-run.py 5000  10.91s user 3.26s system 117% cpu 12.076 total
pip install https://github.com/nornir-automation/nornir/archive/develop.zip
(pip install pydantic==1.3 because it got upgraded to 1.4..)
time python nornir-run.py 5000
(ran three times)
python nornir-run.py 5000  21.00s user 3.45s system 109% cpu 22.282 total
python nornir-run.py 5000  20.47s user 3.12s system 109% cpu 21.575 total
python nornir-run.py 5000  19.82s user 2.88s system 107% cpu 21.108 total

Basically the PR #467, made this twice as slow. I’m guessing that the change was made after checking a task where we were mostly waiting for the network where my example is related to local processing. As the example with concurrent.futures isn’t faster for the network task I would vote to revert #467. The description of Nornir is “Pluggable multi-threaded framework with inventory management to help operate collections of devices”, I think this should perhaps clarify any misconception regarding multiprocessing or threading. If we want to be extra helpful I guess we can insert a comment above the from multiprocessing.dummy import Pool line, I’m not sure that I see it as a big problem though.

/cc @dmfigol

[dbarrosop]

Thanks for this. I will hold the 2.4.0 release until we have clarified this (unless people starts having lots of issues with pydantic, in which case I will revert the PR that introduced this change so we can reevaluate for 2.5.0)

[dmfigol]

I am biased since I submitted PR, but I’d prefer not to revert it unless we have enough evidence it is causing issues. Also, I just realized, that we are in no rush for the release. At first I though that everyone doing pip install nornir will hit pydantic issue, but I just realized that current PyPI version of nornir has pydantic pinned to 0.18.2, so problems with pydantic only affect people using develop branch.