Streamlit doesnot works with Multiprocessing pool
See original GitHub issueSummary
Streamlit apps does not wait for multiprocessing pool to finish even if the Pool.join()
is being used.
Steps to reproduce
In the streamlit python file, create a process pool and wait for it to join.
Expected behavior:
App is not holding for process to join.
Actual behavior:
Explain the buggy behavior you experience when you go through the steps above. If applicable, add screenshots to help explain your problem.
Debug info
- Streamlit version: (get it with
$ streamlit version
) : Streamlit, version 0.64.0 - Python version: (get it with
$ python --version
): Python - Using Conda? PipEnv? PyEnv? Pex?
- OS version: Mac
- Browser version: Chrome latest stable.
Issue Analytics
- State:
- Created 3 years ago
- Reactions:2
- Comments:8 (3 by maintainers)
Top Results From Across the Web
Spawning Process(es) Within a Streamlit Script Can Lead to a ...
Potentially of note, my app spawns several processes (via multiprocessing.pool.Pool) that initiate sometimes longish (10s - 2m) queries on external ...
Read more >multiprocessing.dummy.Pool - Using Streamlit
If I load a pickle file from inside a multiprocessing.dummy.Pool, I need to rerun the streamlit twice every time any code changes.
Read more >App performance caching and multithreading - Using Streamlit
For the caching it works only partially because it does not load every image again, but it does rebuild the UI elements such...
Read more >Streamlit crashes when using Leveldb
LevelDB supports multithread access but it does not support multiprocessing. ... Is there a way to make it work inside streamlit?
Read more >Using Streamlit: how to post a question in the Streamlit forum
Try posting more general Python questions on StackOverflow or a ... Ensure that your example works by running it yourself before posting.
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
I can confirm this issue: We use streamlit behind traefik as a reverse proxy. Using simple-transformers together with caching would cause the application to crash/stop with no error message. Curiously, there was no such issue just using
streamlit run
on Windows. The root cause in our case: simple-transformers uses multiprocessing by default. The solution was simply to disable multiprocessing. (https://simpletransformers.ai/docs/usage/#configuring-a-simple-transformers-model)Just leaving this here for other people hitting this issue, it was very hard to debug. 😃
Does it explain why the error message says that streamlit couldn’t hash a
function
type object, while it is perfectly able to do so whenmultiprocessing
is not involved? I also fail to understand why the error throws an infinite loop and not a finite loop of length the number of sub-interpreters asked (which in my MWE isREPLICATIONS = 1
).Also in my initial MWE the code executed in
starmap
doesn’t use the cached function nor the output of the cached function. Isn’t that strange that there is still an error in that case?