Unify ray discovery during tests

This came up while working on #3898

At the moment, we have two different types of tests:

The ones in ray/test, which will be run with whichever version of ray is installed (this could be the one in the ray/ folder if pip install -e was used, or it could be the one in the site-packages folder, or in the user’s local site-packages folders).

The ones in the subfolders of ray/python/ray, which will run whichever ray version is in the ray/ folder and this will fail if there is no binaries in that folder.

The reason for this difference is the discovery mechanism of pytest: If there is an init.py in the test directory, it will recursively try to go up until it finds the package, see https://docs.pytest.org/en/latest/goodpractices.html the part that says “pytest supports two common test layouts:”).

This behavior is confusing at the moment and should be unified. I’d prefer if the installed version of Ray is always used. In this case we need to get rid of init.py in the test directories, which means we need to put test modules that get imported and test files that get run into separate directories.

cc @richardliaw @ericl @williamma12

[pcmoritz]

Alright, I think we should use --pyargs ray as the short term workaround as @robertnishihara suggests and move on to bazel test as soon as possible.

[richardliaw]

Closed since we moved to Bazel.