Ray actors marked as dead while large file are being written.

What is the problem?

ray 0.8.2

I have the following setup: I have 2 nodes on my driver and many other nodes on remote machines. One of the nodes on my driver is periodically writing large files to disk. When the disk-writer writes the file, I get the following error:

2020-04-29 20:08:58,747#011WARNING worker.py:1058 -- The node with client ID 116cd6d7effde94a120374931760615a6acec931 has been marked dead because the monitor has missed too many heartbeats from it.
2020-04-29 20:08:58,767#011WARNING worker.py:1058 -- A worker died or was killed while executing task ffffffffffffffffee9d776a0100.

Shortly after, my job crashes.

Writing to disk was taking > 30 seconds. I was able to work around this bug by adjusting the num_heartbeats_timeout parameter in _internal_config, so that time needed to mark a node as failed is > 30 sec. But it seems a little strange to be modifying something marked “For testing purposes only”.

I have a few questions:

1. Why are workers marked dead when I’m writing to disk?
2. I eventually write the files to S3; if I upload the objects directly (they’re numpy arrays), will that avoid this issue altogether?
3. As long as disk writes take less time than the heartbeat timeout, am I guaranteed to avoid this problem?

Reproduction (REQUIRED)

Will update in a couple days with a script.

[rkooo567]

Is there any way you can see if the same issue happens in 0.8.4?

[ericl]

Duplicates https://github.com/ray-project/ray/issues/11624