Initializers with the same seed produce different output in Keras 2.7

System information.

• Have I written custom code (as opposed to using a stock example script provided in Keras): yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 2.7.0rc1
• Python version: 3.8
• Bazel version (if compiling from source): N/A
• GPU model and memory: N/A
• Exact command to reproduce:

import numpy as np
import tensorflow as tf

init0 = tf.keras.initializers.RandomUniform(seed=0)
w0 = init0((2, 2))

init1 = tf.keras.initializers.RandomUniform(seed=0)
w1 = init1((2, 2))

print(w0)
print(w1)

assert np.allclose(w0, w1)

Describe the problem.

As far as I understand it, initializers with the same seed should produce the same output. This is how they behaved prior to 2.7. But now passing the same seed to two different initializer instances does not make them produce the same output.

This is likely related to this change mentioned in the TF release notes:

RNG behavior change for all tf.keras.initializers classes. For any class constructed with a fixed seed, it will no longer generate same value when invoked multiple times. Instead, it will return different value, but a determinisitic sequence. This change will make the initialize behavior align between v1 and v2.

However, that seems to suggest that if I create an initializer with a fixed seed, it will always produce the same sequence of numbers (x0 -> x1 -> x2 -> ...). That is not the behaviour observed here; creating two different classes, with the same seed, produces two different sequences. The first class produces x0 -> x1 -> x2 -> ..., but the second produces y0 -> y1 -> y2 -> ....

Describe the current behavior.

Initializers created with the same seed produce different output.

Describe the expected behavior.

Initializers created with the same seed should produce the same output.

Contributing.

• Do you want to contribute a PR? (yes/no): no
• If yes, please read this page for instructions
• Briefly describe your candidate solution(if contributing):

Standalone code to reproduce the issue.

https://gist.github.com/drasmuss/212686552bc5f36a7cd23229f9dc4477

[qlzh727]

Btw, you can take a look for the example in https://www.tensorflow.org/api_docs/python/tf/random/set_seed. There is actually some behavior details.

In the meantime, you can also try:

import numpy as np
import tensorflow as tf

init0 = tf.keras.initializers.RandomUniform(seed=0)
init0._random_generator._force_generator = True
w0 = init0((2, 2))

init1 = tf.keras.initializers.RandomUniform(seed=0)
init1._random_generator._force_generator = True
w1 = init1((2, 2))

print(w0)
print(w1)

assert np.allclose(w0, w1)

[drasmuss]

If you reset the colab runtime then you effectively just have a single initializer (because resetting gets rid of the previous one). The documentation explicitly says that multiple initializers should produce the same sequence with the same seed. You could adjust the documentation to remove that claim entirely, but that’s a pretty significant change in the behaviour.

As a broader point, the benefit of setting the seed to a specific value on an object with randomness is to set that randomness to a reliable, reproducible value. With the current behaviour, the actual seed value passed doesn’t really matter, because all the seed does is fix the RNG to some arbitrary point. For example, consider this code snippet:

w0 = tf.keras.initializers.RandomUniform(seed=0)((2, 2))
w1 = tf.keras.initializers.RandomUniform(seed=0)((2, 2))
w2 = tf.keras.initializers.RandomUniform(seed=1)((2, 2))

There is no effective difference here between setting seed=0 and seed=1. In both cases w0 != w1 and w0 != w2. That seems like unexpected behaviour for a seed.

For a more real-world example, consider the case where I want to build a model that has two layers with the same initial weights in each layer. A natural way to go about that would be the following:

import tensorflow as tf

my_model = tf.keras.Sequential(
    [
        tf.keras.layers.Dense(
            2,
            kernel_initializer=tf.keras.initializers.RandomUniform(seed=0),
            input_shape=(2,),
        ),
        tf.keras.layers.Dense(
            2,
            kernel_initializer=tf.keras.initializers.RandomUniform(seed=0),
        ),
    ]
)

my_model.build()
print(my_model.weights)

I think most users would be pretty surprised to find that that their two layers actually have completely different initial weights, even though they explicitly seeded them to the same value.