Parallel data ingest can fail on malformed data if splitting is unfortunate

See more in the PR #2828.

Details: https://github.com/modin-project/modin/issues/2845#issuecomment-1164484092:

I believe I have (accidentally) found the root cause for this issue while debugging some CI failure.

Imagine the scenario:

1. We have some .csv file which we call pd.read_csv() against
2. That .csv file is a bit incorrect, e.g. some lines in it have too many fields (like in test_read_csv_error_handling())
3. If partition split happens in a way that the very first line in a .csv chunk is a bad one, pandas would happily parse that chunk assuming that it’s other lines being shorter, not that one being invalid
4. Other distributed text logic fails then, because chunks are uneven and their indices could go strange.

I believe this could be relevant for other distributed text file parsers, as we use the same technique almost everywhere.

[vnlitvinov]

I believe I have (accidentally) found the root cause for this issue while debugging some CI failure.

Imagine the scenario:

1. We have some .csv file which we call pd.read_csv() against
2. That .csv file is a bit incorrect, e.g. some lines in it have too many fields (like in test_read_csv_error_handling())
3. If partition split happens in a way that the very first line in a .csv chunk is the bad one, pandas would happily parse that chunk assuming that it’s other lines being shorter
4. Other distributed text logic fails then, because chunks are uneven and their indices could go strange.

I believe this could be relevant for almost all distributed text file parsing, as we use the same technique everywhere.

[vnlitvinov]

I have filed https://github.com/pandas-dev/pandas/issues/47490, let’s see what they respond with.