Enable reproducibility

🚀 Feature request

To enable consistent benchmarking using the transformers library, deterministic behaviour needs to be enforced. This could be a simple option in TrainingArguments, e.g., enforce_reproducibility=True. Currently seeds are set, but randomness still occurs as part of CUDA and the dataloaders.

Motivation

I am the maintainer of a Scandinavian benchmarking library for language models, which uses transformers under the hood. The benchmarking results are always slightly different, however, and this could be resolved in PyTorch as described here. See below for the concrete changes.

Your contribution

To ensure reproducibility, the set_seed function in trainer_utils.py needs to include the following:

import torch
import os

# Enable PyTorch deterministic mode. This potentially requires either the environment
# variable 'CUDA_LAUNCH_BLOCKING' or 'CUBLAS_WORKSPACE_CONFIG' to be set,
# depending on the CUDA version, so we set them both here
os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':16:8'
torch.use_deterministic_algorithms(True)

# Enable CUDNN deterministic mode
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

Furthermore, to enable determinism in PyTorch DataLoaders, the arguments generator and worker_init_fn need to be set. The generator is already set in the transformers library here, so we only need to set the worker_init_fn, as follows:

def seed_worker(_):
    worker_seed = torch.initial_seed() % 2**32
    set_seed(worker_seed)

dataloader = Dataloader(..., worker_init_fn=seed_worker)

[sgugger]

I have not 😃

[saattrupdan]

@hasansalimkanmaz I have not started working on it, no, so if you’ve got the time to look at it then go ahead 😊

Unless @sgugger have already started on it?