It is possible to kill the raylet by writing some bytes to it over TCP.

Start Ray with e.g.,

ray start --head --node-manager-port=12345

Then in a separate process (you can also do this on a separate machine, but then you need to change the host), you can do

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('localhost', 12345))

s.send(1000 * b'asdf')

Then if you look at the raylet logs under /tmp/ray/..., you will see an error like

F0207 21:28:52.615898 2396021632 client_connection.cc:245]  Check failed: read_version_ == RayConfig::instance().ray_protocol_version() 
*** Check failure stack trace: ***
*** Aborted at 1549603732 (unix time) try "date -d @1549603732" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGABRT (@0x7fff5633fb66) received by PID 30417 (TID 0x7fff8ed06380) stack trace: ***
    @     0x7fff564fdf5a _sigtramp

cc @pcmoritz @richardliaw @hartikainen

[raulchen]

This is duplicated with https://github.com/ray-project/ray/issues/3915. We’ve actually seen this error in prod. But we didn’t figure out who was sending the bad bytes. @zhijunfu fixed this by requiring all clients to send a connect_client message before sending other messages. Then we will can ignore this error it it’s from an unknown client. Do you think the fix makes sense? we can open an PR for this.

[robertnishihara]

Ah thanks @raulchen!