Datasets

Description

In discussing with folks at the CCRMA MIR workshop, it seems like there’s a legitimate need for an easy way to access data for development / debugging purposes. I don’t think it makes sense to ship more than one or two tracks in the main librosa package itself, but I’m open to the idea of having some kind of dataset module akin to those found in sklearn, nltk, etc.

I suspect most people will agree that this would be useful in the abstract, but there’s a lot of details to sort out before such a thing could be reasonably implemented. A few points to kick off the discussion:

1. Should this even be a librosa function? e.g., librosa.data.install('gtzan')? Or should it live in an entirely different project (librosa_data)?
2. Where should data be hosted? I could imagine things getting rather large. Would figshare be reasonable?
3. Where should datasets be indexed? Internally to librosa, or should it have a remote index hosted on the web so that we can add new datasets out of cycle with librosa releases?
4. What format should data be stored in? I’m thinking ogg or flac for audio. What about annotations? I’d obviously lean toward jams, but there are some floating issues around collection management that I think should be sorted out before we go down that road. This also adds considerable (but, I believe in the long-run, reasonable) complexity overhead to getting directly at the raw data (via jams objects).
5. How are collections accessed once installed? Unlike standard UCI machine learning datasets, MIR data does not have a nice, fixed-dimensional tabular form in general, so we can’t just have a pair of X/Y arrays. This issue might resolve itself if we go ahead with jams after implementing some collection-management utilities, but it’s worth considering in advance.

(tag @justinsalamon @stefan-balke @jongwook)

[lostanlen]

update: colleagues have started the mirdata project. There is no official release yet, but it seems mature enough already to be considered a solid candidate for us to close this issue, and defer to them for our data-intensive debugging/development/demonstration/education/reproducibility purposes.

https://github.com/mir-dataset-loaders/mirdata

note that mirdata has virtually zero dependencies, if we leave aside those needed for compatibility (six, future) and testing (pytest, testcontainers). It’s distributed in PyPI and the source code has a BSD-3 license. So although i don’t see the need to make mirdata a dependency of librosa, i would be totally in favor of extensively using mirdata in our demos and tutorials for how to use librosa beyond a single audio track (good old Kevin MacLeod …)

[lostanlen]

closed by https://github.com/mir-dataset-loaders/mirdata/pull/150 many thanks to @rabitt, @drubinstein, @magdalenafuentes, @andreasjansson, @keunwoochoi, and everyone who contributed dataset loaders.