High Gamma Dataset Deep4Net results reproduction

I cannot reproduce the results of the Deep4Net with the HGD.

The accuracy should be at around 90% (for subject 1). I am able to reproduce these results with the ShallowNet but not with the DeepNet. Is there anything special to the training, optimization, regularization, etc. of the DeepNet compared to the ShallowNet? @robintibor: any ideas?

I used the same hyperparameters as in the 2017 publication. The only thing I changed in both cases (ShallowNet and DeepNet) is that I used Conv2dWithContraint instead of Conv2d to add extra regularization:

class Conv2dWithConstraint(nn.Conv2d):
    def __init__(self, *args, max_norm=1, dim=0, **kwargs):
        self.max_norm = max_norm
        self.dim = dim
        super(Conv2dWithConstraint, self).__init__(*args, **kwargs)

    def forward(self, x):
        self.weight.data = torch.renorm(
            self.weight.data, p=2, dim=self.dim, maxnorm=self.max_norm
        )
        return super(Conv2dWithConstraint, self).forward(x)

The rest of the code below borrows from the BCICIV2a_example in this repository as well as from the original HGD repo.

Code is available at: https://colab.research.google.com/drive/1tpStIcOFyZNa8Z2ePBxxJztqzvC0qcNl?usp=sharing

[robintibor]

Hm so hyperparams might be scattered among those files: https://github.com/robintibor/braindevel/blob/21f58aa74fdd2a3b03830c950b7ab14d44979045/braindecode/configs/experiments/paper/bci_competition/cnt/deep_4_cnt_net_car.yaml / https://github.com/robintibor/braindevel/blob/21f58aa74fdd2a3b03830c950b7ab14d44979045/braindecode/configs/experiments/4sec_movements/cnt/cnt_4_layers_simple.yaml https://github.com/robintibor/braindevel/blob/21f58aa74fdd2a3b03830c950b7ab14d44979045/braindecode/configs/experiments/4sec_movements/cnt/defaults.yaml Since I cannot see them there, assume we used default learning rate (1e-3) and weight decay 0 for Adam.

In general though, I would recommend to rather try tuning the hyperparams with the new pipeline (training on all data and using cosine annealing), I would expect the overall result to be better.

[martinwimpff]

Hi @bruAristimunha, I closed it by accident 😉 I ran some experiments today and got at least in the right range (86-87% overall accuracy). The gap to ShallowNet remains quite large (overall accuracy 93%). I think one would have to run some more experiments to get a really good configuration. But as we are in roughly the same range, I am okay with closing this issue.

If anyone finds a configuration, that matches the published results, feel free to share it here 😉