Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

FCMA permutation feature selection Acc above chance

See original GitHub issue


I’m using FCMA with Brianiak for my analysis - first thank you for developing this awesome toolbox!

Everything makes sense to me until I started to do the permutation task as mentioned in Wang et al., 2015 for information mapping. Based on my understanding, after I randomized the data, the classification acc (per voxel) I obtain during the feature selection step should be around chance level – just as a sanity check. However, the top voxel’s acc I got for each permutation is around 60%, which is way above chance, and does not make sense to me, suggesting that even if I throw random time series into the clf, the performance is still above chance. In order to make sure that this is replicable, I run FCMA permutation test with the face-scene dataset from the BRAINIAK dataset and got similar result – with the original data, the top voxel acc is around 80%, and for the permutation test, after randomization, the top voxel acc is around 70% (still above chance).

The only change I made to the script is by adding the RandomType argument as following. Note that the top_3000 voxel’s locations I got during permutation test are indeed different from those I got from the original data – they are random, unclustered. The only thing that bothers me is their above chance acc.

raw_data, _, labels = prepare_fcma_data(images, epoch_list, mask, random=RandomType.REPRODUCIBLE)

I’m not sure which step i did wrong that ended up causing this problem, or if I misunderstood the concept and that this above chance performance was expected for permutation test. I’m grateful for any sorts of help, and thank you all in advance!

Thank you very much, Peeta

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

peetalcommented, Jul 29, 2020

Ok that make sense! Thank you @yidawang and @mihaic for the reply! I’ll look into the distribution of voxel-wise acc and also check the clf acc with the full mask. I’ll also try the permutation test with randomized label instead of the 4D data. It will take a while for the permutation to run but I’ll come back here with what I end up having.

manojneurocommented, Aug 26, 2020

@peetal thanks for working on this and finding a workaround. It would be great if you could paste sample code of your workaround here. Thanks.

Read more comments on GitHub >

github_iconTop Results From Across the Web

4.2. Permutation feature importance - Scikit-learn
The permutation feature importance is defined to be the decrease in a model score when a single feature value is randomly shuffled [1]....
Read more >
8.5 Permutation Feature Importance | Interpretable Machine ...
A feature is “unimportant” if shuffling its values leaves the model error unchanged, because in this case the model ignored the feature for...
Read more >
Full correlation matrix analysis (FCMA): An unbiased method ...
With the data-driven feature selection approach described below, ... against chance (50%) was assessed with a permutation test over the ...
Read more >
Selecting Features with Permutation Importance - LinkedIn
Permutation importance quantifies the significance of each feature and therefore can determine an importance value attributable to random noise.
Read more >
brainiak/brainiak - Gitter
In order to make sure that this is replicable, I run FCMA permutation test with the face-scene ... the top voxel acc is...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found