val_dataloader is called twice in each worker

🐛 Bug

I’m trying a LightningDataModule class to manage the data. Using horovod backend, if that matters. I’ve noticed that each rank is calling train_dataloader once, but val_dataloader two times somehow.

To Reproduce

run LIghtning with Dataclass and horovod, add some debug print on when val_dataloader is called

soemthing like

    def train_dataloader(self):
        print(f"\n#####worker {hvd.rank()} of {hvd.size()} creating train_loader\n")
        return load_ds_from_dir(os.path.join(self.path, "train"), self.batch_size)

    def val_dataloader(self):
        print(f"\n#####worker {hvd.rank()} of {hvd.size()} creating val\n")
        return load_ds_from_dir(os.path.join(self.path, "validation"), self.batch_size)

Expected behavior

expect val loader to be called only once…

Environment

* CUDA:
	- GPU:
		- Tesla V100-SXM2-16GB
		- Tesla V100-SXM2-16GB
		- Tesla V100-SXM2-16GB
		- Tesla V100-SXM2-16GB
	- available:         True
	- version:           10.2
* Packages:
	- numpy:             1.19.1
	- pyTorch_debug:     False
	- pyTorch_version:   1.6.0
	- pytorch-lightning: 0.9.0
	- tensorboard:       2.2.0
	- tqdm:              4.46.1
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		- ELF
	- processor:         x86_64
	- python:            3.8.2
	- version:           #1 SMP Fri Apr 20 16:44:24 UTC 2018

[undertherain]

Thanks for considering to update the docs!

[stale[bot]]

This issue has been automatically marked as stale because it hasn’t had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!