Bus error(core dumped) in EdgeDataLoader of unsupervised graphsage example (20M edges)

🐛 Bug

To Reproduce

Steps to reproduce the behavior:

1. In the example code: https://github.com/dmlc/dgl/blob/master/examples/pytorch/graphsage/advanced/train_sampling_unsupervised.py
2. Feed a dataset with 10M nodes and 20M edges
3. Bus error(core dumped) happened in EdgeDataLoader before training and DataLoader in inference.

Expected behavior

Environment

• DGL Version (e.g., 1.0): 0.8.1cu101
• Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3): PyTorch 1.8.1+cu101
• OS (e.g., Linux): Docker
• How you installed DGL (conda, pip, source): pip
• Build command you used (if compiling from source):
• Python version: 3.6.13
• CUDA/cuDNN version (if applicable): 101
• GPU models and configuration (e.g. V100): V100
• Any other relevant information:

Additional context

[yaox12]

This line of code in dataloader will create a shared memory array for shuffling. https://github.com/dmlc/dgl/blob/5ba5106acab6a642e9b790e5331ee519112a5623/python/dgl/dataloading/dataloader.py#L146-L149 When len(train_seeds) > 8M, the shared tensor will run out of Docker’s default shm size 64MB. @jermainewang @BarclayII

[BarclayII]

We can change the code to use shared tensors only when these conditions hold.

That sounds reasonable. I’m not sure how PyTorch shares the tensor to forked subprocesses though: if PyTorch uses shared memory then we are technically still copying the ID tensor into shared memory implicitly.