High Gamma Dataset Deep4Net results reproductionSee original GitHub issue
I cannot reproduce the results of the Deep4Net with the HGD.
The accuracy should be at around 90% (for subject 1). I am able to reproduce these results with the ShallowNet but not with the DeepNet. Is there anything special to the training, optimization, regularization, etc. of the DeepNet compared to the ShallowNet? @robintibor: any ideas?
I used the same hyperparameters as in the 2017 publication. The only thing I changed in both cases (ShallowNet and DeepNet) is that I used Conv2dWithContraint instead of Conv2d to add extra regularization:
class Conv2dWithConstraint(nn.Conv2d): def __init__(self, *args, max_norm=1, dim=0, **kwargs): self.max_norm = max_norm self.dim = dim super(Conv2dWithConstraint, self).__init__(*args, **kwargs) def forward(self, x): self.weight.data = torch.renorm( self.weight.data, p=2, dim=self.dim, maxnorm=self.max_norm ) return super(Conv2dWithConstraint, self).forward(x)
Code is available at: https://colab.research.google.com/drive/1tpStIcOFyZNa8Z2ePBxxJztqzvC0qcNl?usp=sharing
- Created a year ago
- Comments:5 (2 by maintainers)
Top GitHub Comments
Hm so hyperparams might be scattered among those files: https://github.com/robintibor/braindevel/blob/21f58aa74fdd2a3b03830c950b7ab14d44979045/braindecode/configs/experiments/paper/bci_competition/cnt/deep_4_cnt_net_car.yaml / https://github.com/robintibor/braindevel/blob/21f58aa74fdd2a3b03830c950b7ab14d44979045/braindecode/configs/experiments/4sec_movements/cnt/cnt_4_layers_simple.yaml https://github.com/robintibor/braindevel/blob/21f58aa74fdd2a3b03830c950b7ab14d44979045/braindecode/configs/experiments/4sec_movements/cnt/defaults.yaml Since I cannot see them there, assume we used default learning rate (1e-3) and weight decay 0 for Adam.
In general though, I would recommend to rather try tuning the hyperparams with the new pipeline (training on all data and using cosine annealing), I would expect the overall result to be better.
Hi @bruAristimunha, I closed it by accident 😉 I ran some experiments today and got at least in the right range (86-87% overall accuracy). The gap to ShallowNet remains quite large (overall accuracy 93%). I think one would have to run some more experiments to get a really good configuration. But as we are in roughly the same range, I am okay with closing this issue.
If anyone finds a configuration, that matches the published results, feel free to share it here 😉