Unexpected randomness in graphconv prediction

VERSION: 2.1.0

After training a GraphConvModel for a fair number of hours, I am getting some unexpected results when testing the performance of the model. In particular, the training, valid, and test errors are all different when I run the code below. This is surprising since I was careful to set the same seed during the train test split and have not changed the csv file containing the data between training and testing.

If it helps, my original model was constructed using the following code:

TRAINING CODE:

model = GraphConvModel(n_tasks=1, mode='regression',
                       tensorboard=True, model_dir='models/',
                       dropout=0.5, graph_conv_layers=[64,64,64])

Is it possible that dropout is being used in the prediction, leading to the stochastic behaviour? Otherwise, I cannot find anywhere else that randomness has been introduced, and this is quite frustrating since I did not save my datasets to disk, thinking that setting the seed would be sufficient. I have all of the model data, but I doubt that helps without knowing if I am snooping on my test data.

TESTING CODE:

import deepchem as dc
from deepchem.models.tensorgraph.models.graph_models import GraphConvModel

graph_featurizer = dc.feat.graph_features.ConvMolFeaturizer()
loader = dc.data.data_loader.CSVLoader( tasks=['unique_pka'],
                                         smiles_field="CANONICAL_SMILES",
                                        id_field="CMPD_CHEMBLID",
                                        featurizer=graph_featurizer )

dataset = loader.featurize( './unique_pkas.csv' )

splitter = dc.splits.RandomSplitter()
train_dataset, valid_dataset, test_dataset = splitter.train_valid_test_split(dataset, seed=42)

transformers = [
        dc.trans.NormalizationTransformer(transform_y=True, dataset=train_dataset)]

for dataset in [train_dataset, valid_dataset, test_dataset]:
        for transformer in transformers:
                    dataset = transformer.transform(dataset)

model = GraphConvModel.load_from_dir('models')
# model.restore()

train_scores = model.evaluate(
                train_dataset,
                [dc.metrics.Metric(dc.metrics.rms_score),
                 dc.metrics.Metric(dc.metrics.r2_score)]
                )
print('train scores')
print(train_scores)

valid_scores = model.evaluate(
                valid_dataset,
                [dc.metrics.Metric(dc.metrics.rms_score),
                 dc.metrics.Metric(dc.metrics.r2_score)]
                )
print('valid scores')
print(valid_scores)

test_scores = model.evaluate(
                test_dataset,
                [dc.metrics.Metric(dc.metrics.rms_score),
                 dc.metrics.Metric(dc.metrics.r2_score)]
                )
print('test_scores')
print(test_scores)

Thanks for your help!

[peastman]

You’re completely right. train_test_valid_split() was ignoring the seed instead of passing it on to the splitter. I just posted a pull request with the fix. Thanks for spotting this!

[peastman]

Ok, let me look into it and see if I can tell what’s going on.