failed to reduce memory cost by using Dataloader

❓ Questions & Help

I have a big graph whose x is (153364,67) and I create dataset like this

class MyOwnDataset(Dataset):
    def __init__(self, root='./', transform=None, pre_transform=None):
        super(MyOwnDataset, self).__init__(root, transform, pre_transform)

    @property
    def raw_file_names(self):
        return ['some_file_1', 'some_file_2']

    @property
    def processed_file_names(self):
        return ['data.npz']

    def __len__(self):
        return len(self.processed_file_names)


    def _download(self):
        pass

    def _process(self):
        pass

    def get(self, idx=None):
        data = torch.load('data.npz')
        return data

and use dataloader like this: loader = DataLoader([data],batch_size=16).

But when I come to the code:

        for batch in loader:
            optimizer.zero_grad()
            batch = batch.to(dev)
            loss = loss_op(model(batch.x, batch.train_edge_index)[batch.train_mask], batch.y[batch.train_mask])
            loss.backward()
            optimizer.step()

batch.x is still (153364,67) and caused the OOM error, I’m very confused with this bug and looking forward to your reply.

[w21180239]

Yes, you are right. We still need to convert some existing operators to be able to automatically support bipartite graph message passing (this is why we reimplement SAGEConv in the example). You can simply use the SAGEConv operator defined in the example.

Edit: The SAGEConv in nn.conv now also uses the MessagePassing interface.

Awesome, You really are the most enthusiastic man I have ever seem in github!

[rusty1s]

Yes, you are right. We still need to convert some existing operators to be able to automatically support bipartite graph message passing (this is why we reimplement SAGEConv in the example). You can simply use the SAGEConv operator defined in the example.

Edit: The SAGEConv in nn.conv now also uses the MessagePassing interface.