test_udf.py silently failing in CI

Link: https://github.com/modin-project/modin/runs/4288131771?check_suite_focus=true

Example logs:

2021-11-22 15:21:17,206	ERROR worker.py:80 -- Unhandled error (suppress with RAY_IGNORE_UNHANDLED_ERRORS=1): ray::deploy_ray_func() (pid=2408, ip=10.1.0.122)
  File "python\ray\_raylet.pyx", line 536, in ray._raylet.execute_task
  File "D:\a\modin\modin\modin\core\execution\ray\implementations\pandas_on_ray\partitioning\axis_partition.py", line 211, in deploy_ray_func
    result = func(*args)
  File "D:\a\modin\modin\modin\core\dataframe\pandas\partitioning\axis_partition.py", line 160, in deploy_axis_func
    result = func(dataframe, **kwargs)
TypeError: dict_apply_builder() got an unexpected keyword argument 'internal_indices'

2021-11-22 15:21:16,951	ERROR worker.py:80 -- Unhandled error (suppress with RAY_IGNORE_UNHANDLED_ERRORS=1): ray::deploy_ray_func() (pid=2408, ip=10.1.0.122)
  File "python\ray\_raylet.pyx", line 536, in ray._raylet.execute_task
  File "D:\a\modin\modin\modin\core\execution\ray\implementations\pandas_on_ray\partitioning\axis_partition.py", line 211, in deploy_ray_func
    result = func(*args)
  File "D:\a\modin\modin\modin\core\dataframe\pandas\partitioning\axis_partition.py", line 160, in deploy_axis_func
    result = func(dataframe, **kwargs)
  File "D:\a\modin\modin\modin\core\dataframe\pandas\dataframe\dataframe.py", line 1181, in _map_reduce_func
    series_result = func(df, *args, **kwargs)
  File "D:\a\modin\modin\modin\core\storage_formats\pandas\query_compiler.py", line 2445, in <lambda>
    lambda df: pandas.DataFrame(df.apply(func, axis, *args, **kwargs)),
  File "C:\Miniconda\envs\modin\lib\site-packages\pandas\core\frame.py", line 8740, in apply
    return op.apply()
  File "C:\Miniconda\envs\modin\lib\site-packages\pandas\core\apply.py", line 659, in apply
    return self.apply_multiple()
  File "C:\Miniconda\envs\modin\lib\site-packages\pandas\core\apply.py", line 518, in apply_multiple
    return self.obj.aggregate(self.f, self.axis, *self.args, **self.kwargs)
  File "C:\Miniconda\envs\modin\lib\site-packages\pandas\core\frame.py", line 8547, in aggregate
    relabeling, func, columns, order = reconstruct_func(func, **kwargs)
  File "C:\Miniconda\envs\modin\lib\site-packages\pandas\core\aggregation.py", line 88, in reconstruct_func
    "Function names must be unique if there is no new column names "
pandas.core.base.SpecificationError: Function names must be unique if there is no new column names assigned

[mvashishtha]

All the test cases where Ray raises the got an unexpected keyword argument error are working as intended. They are testing cases where we give a list of aggregation functions with repeated names. pandas raises *** pandas.core.base.SpecificationError: Function names must be unique if there is no new column names assigned and we test that Modin does the same. e.g. in test_agg_apply_axis_names we catch and check the exception against pandas: https://github.com/modin-project/modin/blob/26cb6be3b740a9a755e28cdfaaa9a126caa4ca52/modin/pandas/test/dataframe/test_udf.py#L96-L103

I verified that if I remove all usages of the agg_func_except_values, which are supposed to cause the error, we don’t get the ray task errors any more. However, I then get the error from #4828 due to the xfail test cases in pytest modin/pandas/test/dataframe/test_udf.py::test_apply_key_error. We should solve that bug separately.

We should try to figure out how to reduce the noise from ray for the got an unexpected keyword argument, but that’s a separate task and it’s not urgent.

[mvashishtha]

One test case with the error:

pytest "modin/pandas/test/dataframe/test_udf.py::test_agg_apply_axis_names[agg-sum sum-rows]"

I’m looking some more.