Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Swifter using only single core

See original GitHub issue

I am applying swifter to a function which takes several values apart from datetime variable. After running the code I saw it using only a single core (available 6 cores). The data is of size 476k rows. With a single core, it takes about 7.5 minutes.

I added a set_npartitions(16) it improved the processing time to 3.5 minutes but still using a single core.

Any reason why it can’t use all the cores?

Issue Analytics

State:
Created 5 years ago
Comments:13 (7 by maintainers)

Top GitHub Comments

3reactions

raghu1121commented, Apr 25, 2019

I did a test with a different function which in turn calls several functions. The performance on this got even worse. I think this something to do with the function itself. You could give a try with solar.get_altitude(). The size of the sample is about half a million rows. Unfortunately, it is taking about 8.5 mins at 1030it/s. It seems the size of the sample and the number of internal calls has an effect.

Update: Surprisingly allow_dask_on_strings(enable=True) solved the problem.

0reactions

jmcarpenter2commented, Apr 29, 2019

Closing this issue because it seems resolved. Please feel free to re-open if you feel otherwise.