FCMA permutation feature selection Acc above chance

Hello,

I’m using FCMA with Brianiak for my analysis - first thank you for developing this awesome toolbox!

Everything makes sense to me until I started to do the permutation task as mentioned in Wang et al., 2015 for information mapping. Based on my understanding, after I randomized the data, the classification acc (per voxel) I obtain during the feature selection step should be around chance level – just as a sanity check. However, the top voxel’s acc I got for each permutation is around 60%, which is way above chance, and does not make sense to me, suggesting that even if I throw random time series into the clf, the performance is still above chance. In order to make sure that this is replicable, I run FCMA permutation test with the face-scene dataset from the BRAINIAK dataset and got similar result – with the original data, the top voxel acc is around 80%, and for the permutation test, after randomization, the top voxel acc is around 70% (still above chance).

The only change I made to the script is by adding the RandomType argument as following. Note that the top_3000 voxel’s locations I got during permutation test are indeed different from those I got from the original data – they are random, unclustered. The only thing that bothers me is their above chance acc.

raw_data, _, labels = prepare_fcma_data(images, epoch_list, mask, random=RandomType.REPRODUCIBLE)

I’m not sure which step i did wrong that ended up causing this problem, or if I misunderstood the concept and that this above chance performance was expected for permutation test. I’m grateful for any sorts of help, and thank you all in advance!

Thank you very much, Peeta

[peetal]

Ok that make sense! Thank you @yidawang and @mihaic for the reply! I’ll look into the distribution of voxel-wise acc and also check the clf acc with the full mask. I’ll also try the permutation test with randomized label instead of the 4D data. It will take a while for the permutation to run but I’ll come back here with what I end up having.

[manojneuro]

@peetal thanks for working on this and finding a workaround. It would be great if you could paste sample code of your workaround here. Thanks.