DOC: Improve doc of "threshold" in permutation cluster funcs

Several cluster permutation functions in the stats module have a threshold parameter, for example: mne.stats.spatio_temporal_cluster_1samp_test. See screenshot below for a quick view of the param:

I think we should improve the documentation in several ways:

1. Seeing “float” as a possible input, people may confuse this threshold with an “alpha” level and input something like the conventionally used p-thresholds like 0.001 or 0.05
2. Instead, a “t” (or “F”) threshold is the expected input. In some tutorials, we can see how to use it, for example here and here: It’s something like this --> threshold = stats.distributions.t.ppf(1 - alpha, n_subjects - 1) … yet the tutorials also quote other options like putting a - before the threshold or dividing it by zero (either the threshold, or the “alpha”) in brackets: I find this whole thing confusing: When is a users supposed to use which method? --> Overall, most people (including myself) will think in terms of alpha levels (p-value-thresholds), rather than t-values: We should make this easy
3. in mne.stats.ttest_1samp_no_p There are some further notes on this that seem out of place: thresh = -scipy.stats.distributions.t.ppf(p_thresh, n_samples - 1) / 2. --> in this function, no threshold parameter is even present 🙂

I am happy to make the doc changes if you all agree and if somebody helps me figure this out.

TL;DR: We should clearly state how people get from an idea like “I want to do a one-sided t-test on the upper tail with a p-value threshold of 0.05” to a value to pass to the threshold parameter.

cc @nomisciri

[hoechenberger]

@sappelhoff

I’m now adding a plot like the following to our reports

This makes it easier to understand which time points were used to form clusters (I will next work on spatio-temporal tests, where instead of a trace I will probably show some kind of heatmap)

See https://output.circle-artifacts.com/output/job/af14b1ea-65f5-46c3-8a44-b7cdaa0710c2/artifacts/0/reports/ERP_CORE/sub-average_ses-N400_task-N400_report.html#global10

The respective PR is at https://github.com/mne-tools/mne-bids-pipeline/pull/552

[hoechenberger]

do you have links to the workshop material? Perhaps I can learn from that as well – in case nobody has a straight forward answer to my question above.

We don’t cover the sign of the threshold there, I’m afraid …

Your question was about the sign, right?

It depends on the tail of the distribution you want to look at. For example, this is what we do when the user does not provide a custom threshold and we calculate our own default:

https://github.com/mne-tools/mne-python/blob/cc86f163126ad1aa950f1909062abe5e9b1888c5/mne/stats/cluster_level.py#L1035-L1036

In case the user does provide a threshold, we have a fail-safe built in that checks for reasonable combinations of threshold and tail:

https://github.com/mne-tools/mne-python/blob/cc86f163126ad1aa950f1909062abe5e9b1888c5/mne/stats/cluster_level.py#L846-L849

And this is the reason why in the one example we have to flip the sign of the t value:

Let’s assume we have 20 participants and want to compare a within-subjects design; to form our initial clusters, we want to find a t-value that separates the most extreme 7% of the expected t-values from the other 93%.

(I picked the odd number 7% here on purpose as to avoid confusion with the “alpha level” which we’ll later use to determine significance of the clusters, and which is typically set at 5%).

We don’t have a clear justification to believe the effect will only go into one specific direction; so we need to perform a two-sided test. This means we need to partition those 7% extreme values into two times 3.5% at the extremes:

In [8]: from scipy.stats import t

In [9]: t.ppf(0.07 / 2, 20-1)
Out[9]: -1.91999159718071

Now this is a negative value because the distribution is centered on 0 and we’re getting the t-value corresponding to the “leftmost” (smallest) 3.5% of the distribution!

But the “fail-safe” in our code won’t allow us to pass a negative threshold together with a tail=0 (two-sided test)

So we have two potential solutions here:

1. We simply multiply by -1, as in the example you quoted
2. Instead of getting the t-value of the smallest 3.5%, we could get the value of the largest 96.5%:

   In [10]: t.ppf(1 - 0.07 / 2, 20-1)
   Out[10]: 1.9199915971807093
   

I agree that in our example, 1. is not very clear, and we should use 2. instead.

Does this help you a little?