Relabeling Modin Frame loses partitions shape cache

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Any
• Modin version (modin.__version__): a3ddf2f
• Python version: 3.7.5
• Code we can use to reproduce:

import modin.pandas as pd

df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})

modin_frame = df._query_compiler._modin_frame
src_cache = modin_frame._partitions[0][0]._length_cache

modin_frame.columns = ["c", "d"]
relabeled_cache = modin_frame._partitions[0][0]._length_cache

assert (
    src_cache == relabeled_cache
), f"src: {src_cache} | relabeled: {relabeled_cache}"

Output:

Traceback (most recent call last):
  File "tst3.py", line 13, in <module>
    ), f"src: {src_cache} | relabeled: {relabeled_cache}"
AssertionError: src: 3 | relabeled: None

Describe the problem

Setting new axis labels should not change partitioning, so the shape of partitions and so its cache should be untouched.

The cache is losing because propagating new labels into partitions (performed by PandasFrame.synchronize_labels) done just like regular function applying: https://github.com/modin-project/modin/blob/a3ddf2f01163a312416d2a8bc456ba9582ae9b4d/modin/engines/base/frame/data.py#L360-L368 This doesn’t give any hints to partition that the shape will be unchanged, so it resets the shape cache.

It seems that there should be some mechanism (for example different apply-function, or a parameter for the existed one) telling partition that it can preserve shape cache.

[devin-petersohn]

@YarShev that is what should happen in this commit. In master, all 19 columns will be on one worker because minimum size is 32.

[mvashishtha]

I think #3662 fixed this.