importing modin result in latest arrow 6.0 hang

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04 LTS Windows 10

• Modin version (modin.__version__): 0.12.1

• Python version: Python 3.9.1

• Code we can use to reproduce:

Install latest modin and arrow with:

conda create -n test -c conda-forge modin-all pyarrow=6.0.1

Then use the code to reproduce.

# importing modin result in hang for arrow iterator
import modin.pandas as pd
import pyarrow.dataset as ds

dfile = 'veterans_lung_cancer.csv'
data = ds.dataset(dfile, format='csv')
rb_iter = iter(data.to_batches())
next(rb_iter)

Also attach the code:

modin_arrow_hang.tar.gz

Describe the problem

The code hang with arrow 6.0, but for arrow 4.0 it’s ok. If modin is removed, everything is fine.

[devin-petersohn]

I have opened an issue here: https://issues.apache.org/jira/browse/ARROW-15362

[devin-petersohn]

Ok, I was able to reproduce this without Modin:

import os
import pyarrow.dataset as ds

os.environ["OMP_NUM_THREADS"] = "1"  # offending line

dfile = 'veterans_lung_cancer.csv'
data = ds.dataset(dfile, format='csv')
rb_iter = iter(data.to_batches())
next(rb_iter)  # hangs

This probably needs to be raised in pyarrow because it is not something we could fix in Modin. Thanks @xwu99 for finding/reporting this!