Error reading big CNT file

MNE Group: I tried to read several cnt files of different sizes using mne.io.read_raw_cnt.

data = mne.io.read_raw_cnt('test.cnt', preload=True, montage=None, stim_channel=True)

If the cnt file is lower than 2G, the file can be successfully loaded. However, if the file size exceeds 2G, I will receive the following error:

<ipython-input-25-acf0568a07e0>:1: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  data = mne.io.read_raw_cnt('test.cnt', preload=True, montage=None, stim_channel=True)
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-25-acf0568a07e0> in <module>
----> 1 data = mne.io.read_raw_cnt('test.cnt', preload=True, montage=None, stim_channel=True)

~/.local/lib/python3.5/site-packages/mne/io/cnt/cnt.py in read_raw_cnt(input_fname, montage, eog, misc, ecg, emg, data_format, date_format, preload, stim_channel, verbose)
    176     return RawCNT(input_fname, montage=montage, eog=eog, misc=misc, ecg=ecg,
    177                   emg=emg, data_format=data_format, date_format=date_format,
--> 178                   preload=preload, stim_channel=stim_channel, verbose=verbose)
    179
    180

~/.local/lib/python3.5/site-packages/mne/io/cnt/cnt.py in __init__(self, input_fname, montage, eog, misc, ecg, emg, data_format, date_format, preload, stim_channel, verbose)
    430         input_fname = path.abspath(input_fname)
    431         info, cnt_info = _get_cnt_info(input_fname, eog, ecg, emg, misc,
--> 432                                        data_format, _date_format, stim_channel)
    433         last_samps = [cnt_info['n_samples'] - 1]
    434         _check_update_montage(info, montage)

~/.local/lib/python3.5/site-packages/mne/io/cnt/cnt.py in _get_cnt_info(input_fname, eog, ecg, emg, misc, data_format, date_format, stim_channel_toggle)
    293         if stim_channel_toggle:
    294             data_format = 'int32' if n_bytes == 4 else 'int16'
--> 295             annot = _read_annotations_cnt(input_fname, data_format=data_format)
    296             events = (np.stack((annot.onset * sfreq,
    297                                 annot.duration * sfreq,

~/.local/lib/python3.5/site-packages/mne/io/cnt/cnt.py in _read_annotations_cnt(fname, data_format)
     68
     69     with open(fname, 'rb') as fid:
---> 70         teeg = _read_teeg(fid, teeg_offset=event_table_pos)
     71
     72     event_parser = _get_event_parser(event_type=teeg.event_type)

~/.local/lib/python3.5/site-packages/mne/io/cnt/_utils.py in _read_teeg(f, teeg_offset)
     26     teeg_parser = Struct('<Bll')
     27
---> 28     f.seek(teeg_offset)
     29     return Teeg(*teeg_parser.unpack(f.read(teeg_parser.size)))
     30

OSError: [Errno 22] Invalid argument

Also: If I use:

data = mne.io.read_raw_cnt('test.cnt', preload=True, montage=None)

I got different error:

<ipython-input-30-dbbb56936797>:1: FutureWarning: The parameter `stim_channel` controlling the stim channel synthesis has not been specified. In 0.18 it defaults to True but will change to False in 0.19 (when no stim channel synthesis will be allowed) and be removed in 0.20; migrate code to use `stim_channel=False` and :func:`mne.events_from_annotations` or set `stim_channel=True` to avoid this warning.
  data = mne.io.read_raw_cnt('test.cnt', preload=True, montage=None)
<ipython-input-30-dbbb56936797>:1: RuntimeWarning: Could not define the number of bytes automatically. Defaulting to 2.
  data = mne.io.read_raw_cnt('test.cnt', preload=True, montage=None)
Reading 0 ... 4528779  =      0.000 ...  4528.779 secs...
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-30-dbbb56936797> in <module>
----> 1 data = mne.io.read_raw_cnt(test.cnt', preload=True, montage=None)

~/.local/lib/python3.5/site-packages/mne/io/cnt/cnt.py in read_raw_cnt(input_fname, montage, eog, misc, ecg, emg, data_format, date_format, preload, stim_channel, verbose)
    176     return RawCNT(input_fname, montage=montage, eog=eog, misc=misc, ecg=ecg,
    177                   emg=emg, data_format=data_format, date_format=date_format,
--> 178                   preload=preload, stim_channel=stim_channel, verbose=verbose)
    179
    180

~/.local/lib/python3.5/site-packages/mne/io/cnt/cnt.py in __init__(self, input_fname, montage, eog, misc, ecg, emg, data_format, date_format, preload, stim_channel, verbose)
    439         data_format = 'int32' if cnt_info['n_bytes'] == 4 else 'int16'
    440         self.set_annotations(
--> 441             _read_annotations_cnt(input_fname, data_format=data_format))
    442
    443     @verbose

~/.local/lib/python3.5/site-packages/mne/io/cnt/cnt.py in _read_annotations_cnt(fname, data_format)
     68
     69     with open(fname, 'rb') as fid:
---> 70         teeg = _read_teeg(fid, teeg_offset=event_table_pos)
     71
     72     event_parser = _get_event_parser(event_type=teeg.event_type)

~/.local/lib/python3.5/site-packages/mne/io/cnt/_utils.py in _read_teeg(f, teeg_offset)
     26     teeg_parser = Struct('<Bll')
     27
---> 28     f.seek(teeg_offset)
     29     return Teeg(*teeg_parser.unpack(f.read(teeg_parser.size)))
     30

OSError: [Errno 22] Invalid argument

Here is my mne info

mne.sys_info()
Platform:      Linux-4.4.0-151-generic-x86_64-with-Ubuntu-16.04-xenial
Python:        3.5.2 (default, Nov 12 2018, 13:43:14)  [GCC 5.4.0 20160609]
Executable:    /usr/bin/python3
CPU:           x86_64: 12 cores
Memory:        Unavailable (requires "psutil" package)
mne:           0.18.1
numpy:         1.16.3 {blas=openblas, lapack=openblas}
scipy:         1.3.0
matplotlib:    3.0.1 {backend=TkAgg}

sklearn:       0.19.1
nibabel:       Not found
mayavi:        Not found
cupy:          Not found
pandas:        0.22.0
dipy:          Not found

[AimerLee]

Since the CNT file is huge, which means the samples section in the CNT file (ref this: Structure of CNT file ) is also huge.

And mne choose to read the event table position from the SETUP section, which read as a long (32 bit) integer.

with open(fname, 'rb') as fid:
        ....
        fid.seek(SETUP_EVENTTABLEPOS_OFFSET)
        (event_table_pos,) = np.frombuffer(fid.read(4), dtype='<i4')`

As a result, the maximum length of the CNT file mne supports is equals to :

maximum_seconds = (np.iinfo(np.int32).max - 900 - 75 * n_channels) / sfreq / n_channels / 4

“900” is the size of the SETUP section, 75*n_channels is the size of ELECTLOC, and “4” is the bytes each sample point takes.

For the situation n_channels=62 and sfreq=1000, the result is 8659 seconds (about 2.4 hours). When the length of your CNT file is too long that the value of EVENT_TABLE_POS will overflow.

To solve this problem, you need to change the way to obtain the value of EVENT_TABLE_POS: calculate it using follow formula instead of read it from SETUP section:

--        # fid.seek(SETUP_EVENTTABLEPOS_OFFSET)         # Line 66
--        # (event_table_pos,) = np.frombuffer(fid.read(4), dtype='<i4')  # Line 67

++        fid.seek(864)
++        (n_samples,) = np.frombuffer(fid.read(4), dtype='<i4')
++        event_table_pos = 900 + 75 * int(n_channels) + 4 * int(n_channels) * int(n_samples)

--          event_offset = np.fromfile(fid, dtype='<i4', count=1)[0]     # Line 242
++        event_offset = 900 + 75 * int(n_channels) + 4 * int(n_channels) * int(n_samples)

[sKinreich]

Hi agramfort,

Thank you for your reply !!! I’m using version 0.22 of MNE. How can I share the file privately ? can you send me your email address ? ([e-mail]) Thanks you again for any help.

Sivan