sqlite3.OperationalError: too many SQL variables

I am using ivadomed (git-master-97b5772374660b12895f7a458941f468090e8bf8*)

I run into the above error when trying to train on the spine generic dataset. Don’t really know what’s causing that.

Traceback (most recent call last):
  File "/home/etetteh/venv_ivadomed/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/home/etetteh/venv_ivadomed/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: too many SQL variables

[mariehbourget]

There would be one thing to fix in the config file in contrast_params. The contrasts in BIDS is always the last part of the filename after the last underscore, so “T1w_MTS” and “MTon_MTS” should be replaced by “MTS” only in:

        "contrast_params": {
            "training_validation": ["T1w", "T1w_MTS", "T2w", "T2star", "MTon_MTS"],
            "testing": ["T1w", "T1w_MTS", "T2w", "T2star", "MTon_MTS"],
            "balance": {}
        },

Other than that, everything looks fine.

I would suggest to fix the contrast_params and try the training again. However, I doubt that this is what is causing the issue with the pybids indexing here.

If the training fails again, I would recommend trying locally on your computer just to see if you get the same error as in compute canada.

[dyt811]

Okay, current theory on this is likely related to the SQLite database being built under BIDSLayout (a package we use) having a hard limit on SQLITE_MAX_COLUMN, as elaborated a bit here: https://stackoverflow.com/a/67673249. We will likely needs to check SQLite version and its limit and see if there is a way to recompile SQLite with higher limit to see if it gets around this issue. I just requested Compute Canada credential and will try to replicate the problems.

AFAIK, this is a lower library issue meaning we won’t be able to address it just by making code change on our side. More investigation will be required once I get my CC access.