Streamlit doesnot works with Multiprocessing pool

Summary

Streamlit apps does not wait for multiprocessing pool to finish even if the Pool.join() is being used.

Steps to reproduce

In the streamlit python file, create a process pool and wait for it to join.

Expected behavior:

App is not holding for process to join.

Actual behavior:

Explain the buggy behavior you experience when you go through the steps above. If applicable, add screenshots to help explain your problem.

Debug info

• Streamlit version: (get it with $ streamlit version) : Streamlit, version 0.64.0
• Python version: (get it with $ python --version): Python
• Using Conda? PipEnv? PyEnv? Pex?
• OS version: Mac
• Browser version: Chrome latest stable.

[salmenf]

I can confirm this issue: We use streamlit behind traefik as a reverse proxy. Using simple-transformers together with caching would cause the application to crash/stop with no error message. Curiously, there was no such issue just using streamlit run on Windows. The root cause in our case: simple-transformers uses multiprocessing by default. The solution was simply to disable multiprocessing. (https://simpletransformers.ai/docs/usage/#configuring-a-simple-transformers-model)

Just leaving this here for other people hitting this issue, it was very hard to debug. 😃

[durandg12]

Does it explain why the error message says that streamlit couldn’t hash a function type object, while it is perfectly able to do so when multiprocessing is not involved? I also fail to understand why the error throws an infinite loop and not a finite loop of length the number of sub-interpreters asked (which in my MWE is REPLICATIONS = 1).

Also in my initial MWE the code executed in starmap doesn’t use the cached function nor the output of the cached function. Isn’t that strange that there is still an error in that case?