Reproducibility issue: `ray.init()` messes up `random.seed(SEED)`

What is the problem?

Simply running ray.init() screws up seeding performed prior to that when using random standard library. Interestingly enough, this does not happen with np.random.

ray: 0.8.6 python: 3.8.3 OS: macOS 10.15.4

Reproduction

Code:

import random
import ray

SEED = 42

random.seed(SEED)
rand1 = random.random()

random.seed(SEED)

ray.init()

rand2 = random.random()

print("rand1:", rand1)
print("rand2:", rand2)

Output:

# ray initialization output...
rand1: 0.6394267984578837
rand2: 0.025010755222666936

[dburov190]

Thanks for quick responses, guys!

A quick google search tells me that it’s impossible to save and restore seeds (or a bad idea), and even though you can save and restore the state (https://stackoverflow.com/a/32172816), it seems the “most correct” way is to have your own random-number generator that does not affect the global state of random.random, numpy.random etc.

So, I guess, something like

import random

def new_port():
    custom_rng = random.Random()
    return custom_rng.randint(10000, 65535)

SEED = 42

random.seed(SEED)
rand1 = random.random()

random.seed(SEED)

port_number = new_port()

rand2 = random.random()

print("rand1:", rand1)
print("rand2:", rand2)
print("port_number:", port_number)

with the output:

rand1: 0.6394267984578837
rand2: 0.6394267984578837
port_number: 39199 # this will be different on each execution

This would completely solve the issue with seeds in the main process (that calls ray.init); I’m not sure what’s gonna happen within the worker processes… but that, to me, seems less of an issue since when you wrote your serial code you were not relying on workers anyway (and the parallelizable bits of code are most likely deterministic; if not, like in MCMC, well, it’s your job to explicitly select seeds that you want for each worker).

[dburov190]

I would assume so… this is at least a very non-trivial behavior, and should be warned about?

Edit: the issue is that a typical code development cycle is: (1) write serial code (2) make sure everything is reproducible, and the results are as expected (3) try to make faster by parallelization if needed

So, in (3) after you use some framework to speed up the code, you expect the result to be the same, and this is an unpleasant surprise… (: