cannot instantiate sfaira universe when sfaira is not installed in editable mode

When running an editable install of the latest release branch of sfaira u = sfaira.data.Universe(data_path="/some/empty/dir/") runs without problems. if the same is done on a non-editable install (pip install sfaira), the below error is raised.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-683a72d34073> in <module>
----> 1 sfaira.data.Universe(data_path="/home/leander.dony/sfaira_data_test")

~/.local/lib/python3.8/site-packages/sfaira/data/dataloaders/super_group.py in __init__(self, data_path, meta_path, cache_path, exclude_databases)
     29         """
     30         dsgs = [
---> 31             DatasetSuperGroupLoaders(
     32                 data_path=data_path,
     33                 meta_path=meta_path,

~/.local/lib/python3.8/site-packages/sfaira/data/dataloaders/loaders/super_group.py in __init__(self, data_path, meta_path, cache_path)
     37                     if path_dsg is not None:
     38                         try:
---> 39                             dsg = DatasetGroupDirectoryOriented(
     40                                 file_base=path_dsg,
     41                                 data_path=data_path,

~/.local/lib/python3.8/site-packages/sfaira/data/dataloaders/base/dataset_group.py in __init__(self, file_base, data_path, meta_path, cache_path)
    767                         datasets.extend(datasets_f)
    768 
--> 769         keys = [x.id for x in datasets]
    770         super().__init__(datasets=dict(zip(keys, datasets)), collection_id=collection_id)
    771 

~/.local/lib/python3.8/site-packages/sfaira/data/dataloaders/base/dataset_group.py in <listcomp>(.0)
    767                         datasets.extend(datasets_f)
    768 
--> 769         keys = [x.id for x in datasets]
    770         super().__init__(datasets=dict(zip(keys, datasets)), collection_id=collection_id)
    771 

~/.local/lib/python3.8/site-packages/sfaira/data/dataloaders/base/dataset.py in id(self)
   1710             return self._id
   1711         else:
-> 1712             raise AttributeError(f"Dataset ID was not set in dataloader in {self.doi_main}, please ensure the "
   1713                                  f"dataloader constructor of this dataset contains a call to self.set_dataset_id()")
   1714 

AttributeError: Dataset ID was not set in dataloader in None, please ensure the dataloader constructor of this dataset contains a call to self.set_dataset_id()

[Zethson]

Okay, I think I found the problem: when sfaira is installed just using pip, all .yaml or .tsv files are skipped. @Zethson do you know how we can force pip to install all files that we have in the package, not just .py files?

Yeah. Been there done that. This is by the way builtin with Poetry 😃

The proper way to do this: https://packaging.python.org/guides/using-manifest-in/ But I never really managed to make it work although it should be trivial. But my attempts are quite in the past.

Here’s some old shit code of mine: https://github.com/Zethson/promoe/blob/master/setup.py#L67 The walker function is in the same setup.py file. This works for sure.

For both solutions note: all files specified by the package_data and data_files setup() arguments

so don’t forget package_data in setup.py (see my old code)

[davidsebfischer]

Thanks a lot Lukas! I’ll give the manifest a try. @davidsebfischer do you think recursive-include sfaira/data/dataloaders/loaders/d* *.tsv *.yaml ie. any tsv or yaml file under sfaira/data/dataloaders/loaders/d* should cover any non-python file that is in sfaira? or is there more (eg. on the ontologies or genome side)? i remember we had some csv files in there as well at some point?

recursive-include sfaira/data/dataloaders/loaders/d* *.tsv *.yaml should be enough! Ontologies and genomes both directly cache from public servers now to make this transparent!