New "associations" metadata

Add a new attribute to BIDSFile with a list of files associated to this particular BIDSFile. Files listed here are, for instance:

• Events files and physio files corresponding to a particular functional run
• bvecs/bvals tables corresponding to a dwi image
• Any valid path with IntendedFor metadata pointing to this hit (e.g. fieldmaps).

The idea would be to minimize subsequent queries to the layout, and having a quick glimpse at the involvement of this hit in the context of the full dataset.

If this feature can be time consuming, then it could be protected by a return_associations argument to get, which would be False by default.

What do you guys think? I could work on this if you see it useful.

Proposed associations

• BOLD runs:

  • events, stim, physio.
  • fieldmaps/EPIs with IntendedFor pointers
  • SBRefs with matching names (i.e. sub-01_sbref.nii.gz would match all tasks, although this is unclear in the specs at the moment) or IntendedFor field (this IntendedFor is not in the spec at the moment, but I would submit a PR to have it).
  • other echoes of ME-EPI

• DWI runs:

  • bvecs and bvals
  • Matching B0s
  • Matching SBRefs
  • fieldmaps/EPIs with IntendedFor pointers

• Multiecho anats:

  • Matching echoes.

[tyarkoni]

@oesteban okay now that I have a mostly working SQLAlchemy-backed branch of pybids (the variable ingestion stuff is currently broken, but all the core layout tests pass), I’ve thought about this a bit more, and I think it will actually be quite easy to implement using the DB, without needing to worry (yet) about all the I/O stuff I mentioned above. It’s straightforward to treat this purely at a database level. I’ll probably just create a FileAssociation join table that links any two files. Then there will be a single mapping function that executes immediately after the current indexing process is done, adding all associations between files. These will be queryable via a bidirectional .associations property on the model.

It would be helpful if you could flesh out as many of the association-defining rules you can think of. The main source of potential ambiguity/problems I can see is inheritance—it feels like it could get messy if we associate a root-level file with 600 files that inherit from it. On the other hand, it would still be good to register this association at DB level, as that would make walking up the tree unnecessary past the initial indexing process. So maybe we can have a closed set of relationship types or indicators, with one binary column being, say, inherited, where True would imply that the file is only associated via inheritance, and False means there’s a direct relationship.

[oesteban]

Hi @tyarkoni, sorry for my slow response testing this. I’ve found a use case that needs revision:

>>> bold = layout.get(suffix='bold', extension=['.nii', '.nii.gz'])
>>> # Get associations of the ``phasediff`` informer (index [-1])
>>> bold[0].get_associations(kind='InformedBy')[-1].get_associations()
[<BIDSJSONFile filename='/home/oesteban/Data/sdcflows-tests/testdata/sub-HCP101006/fmap/sub-HCP101006_phasediff.json'>,
 <BIDSImageFile filename='/home/oesteban/Data/sdcflows-tests/testdata/sub-HCP101006/func/sub-HCP101006_task-rest_dir-LR_bold.nii.gz'>,
 <BIDSImageFile filename='/home/oesteban/Data/sdcflows-tests/testdata/sub-HCP101006/func/sub-HCP101006_task-rest_dir-LR_sbref.nii.gz'>,
 <BIDSImageFile filename='/home/oesteban/Data/sdcflows-tests/testdata/sub-HCP101006/func/sub-HCP101006_task-rest_dir-RL_bold.nii.gz'>,
 <BIDSImageFile filename='/home/oesteban/Data/sdcflows-tests/testdata/sub-HCP101006/func/sub-HCP101006_task-rest_dir-RL_sbref.nii.gz'>]

What I’m missing here is one <BIDSJSONFile filename='/home/oesteban/Data/sdcflows-tests/testdata/sub-HCP101006/fmap/sub-HCP101006_magnitude.nii.gz'>, probably of kind "MetadataFrom" or "InformedBy" ?

That would allow reconstructing the whole metadata graph to set up distortion correction:

bold -> phasediff (via IntendedFor) -> magnitude{1,2,} (via hierarchy)