Working with rank-deficient EEGLAB data

So a colleague of mine was handed a set of files that had been pre-processed in EEGLAB. The data was read via mne.read_epochs_eeglab(). When computing the noise covariance via mne.compute_covariance() and visualizing the results via Covariance.plot(), we noticed that the data looked a bit odd. Turns out that during preprocessing, artifact rejection via ICA had been carried out, and some ICs had been removed. So the data was now rank-deficient. However, MNE didn’t “pick this up” – in fact, mne.compute_rank() assumed the rank was full even with rank=None to force an actual estimation. And according to raw.info, obviously, the rank was full as well.

So I’m wondering how to best deal with this now. We could certainly pass rank=dict(eeg=true_rank) to all functions that require a rank input. But this seems cumbersome and error-prone, and I’m not sure if there are other places where this situation could lead to errors? I would prefer just being able to do something like, raw.rank = true_rank; or, even better, the EEGLAB import should do something of that sort. A way to detect rank deficiency after ICA is simply by checking if the number of sensors is equal to the number of ICs in the EEG structure:

data = scipy.io.loadmat('myfile.set')
n_comps, n_sensors = data['EEG']['icaweights'][0, 0].shape
if n_comps == n_sensors:
    print('Data has full rank')
else:
    print(f'Data has rank {n_comps}')

(Of course this only works if no (avg) reference has been set)

Any opinions on that?

cc @cbrnr

[agramfort]

what do you mean such data? EEGLAB data which contain an ICA solution?

I would suggest to people to look at their covariance

[larsoner]

I didn’t understand how that PR could fix it until I looked back at…

EEGLAB data noise cov estimated with rank=None

… I realized that I probably misunderstood this plot. Did you have some form of regularization on (e.g., shrunk) during estimation? If so it would explain why the first shelf is only 1e-2 or so below the real data (regularization inflated the eigenvalues). Can you do the same plot of the noise covariance estimated with empirical? You will probably see a much larger shelf at 48 (at least 1e-6 below the largest eigenvalue), and hence using a rtol vs atol like in #7736 could indeed help.