[Core][Bug] dask_on_ray_1tb_sort is flaky

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Core

What happened + What you expected to happen

https://console.anyscale.com/o/anyscale-internal/projects/prj_2xR6uT6t7jJuu1aCwWMsle/clusters/ses_GrXJdM9i4sBUJ2Vitg1de7xR?command-history-section=command_history the latest run failed with object location not found error.

Trial 0 start
Traceback (most recent call last):
  File "dask_on_ray/dask_on_ray_sort.py", line 209, in <module>
    file_path=args.file_path)
  File "dask_on_ray/dask_on_ray_sort.py", line 130, in trial
    10, npartitions=-1))
  File "/home/ray/anaconda3/lib/python3.7/site-packages/dask/dataframe/core.py", line 1096, in head
    return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/dask/dataframe/core.py", line 1130, in _head
    result = result.compute()
  File "/home/ray/anaconda3/lib/python3.7/site-packages/dask/base.py", line 288, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/dask/base.py", line 571, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/util/dask/scheduler.py", line 182, in ray_dask_get
    result = ray_get_unpack(object_refs, progress_bar_actor=pb_actor)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/util/dask/scheduler.py", line 475, in ray_get_unpack
    computed_result = get_result(object_refs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/util/dask/scheduler.py", line 463, in get_result
    return ray.get(object_refs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1740, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError: ray::dask:('head-1000-10-sort_index-83dded8a012efbd64d576d110747e40c', 0) (pid=1883, ip=10.0.2.136)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::dask:('head-partial-10-sort_index-83dded8a012efbd64d576d110747e40c', 5) (pid=1884, ip=10.0.2.136)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::dask:('sort_index-83dded8a012efbd64d576d110747e40c', 5) (pid=1884, ip=10.0.2.136)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::dask:('simple-shuffle-b816d627453488d9321c08feb88fb166', 5) (pid=196, ip=10.0.2.85)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.ObjectFetchTimedOutError: Failed to retrieve object a35cf0632d2a65124d4edd08ecb54563bb6836ec0200000002000000. To see information about where this ObjectRef was created in Python, set the environment variable RAY_record_ref_creation_sites=1 during `ray start` and `ray.init()`.

Fetch for object a35cf0632d2a65124d4edd08ecb54563bb6836ec0200000002000000 timed out because no locations were found for the object. This may indicate a system-level bug.

Versions / Dependencies

N/A

Reproduction script

N/A

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[rkooo567]

What about we decrease the timeout and see if it fails more often? (maybe like 3 minutes?)

[stephanie-wang]

Yeah my guess it is either 3 or there is a bug in the error reporting.

On Tue, Jan 4, 2022, 6:17 PM SangBin Cho @.***> wrote:

So, it seems like there are 4 times locations are not found. I am trying to figure out if it is a bug or due to some resources limit (e.g., high cpu or slow network)

1. OBOD is not updated -> shouldn’t happen because tasks are scheduled after deps are ready
2. Pull is already cancelled -> I don’t exactly know when this happens, but my impression is it shouldn’t happen in Dask on Ray
3. OBOD subscription is slow -> Could happen when the driver or raylet has high CPU usage
4. Objects are lost -> didn’t happen based on logs

So, it is highly likely case 3?

— Reply to this email directly, view it on GitHub https://github.com/ray-project/ray/issues/21346#issuecomment-1005242650, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATREBFP6P2W2U2SBDBNYBDUUN5YDANCNFSM5LFXXNNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>