Taking multi-channel seriously

Most of librosa only supports monophonic audio. But for many of the analyses we’d like to do, stereo or multi-channel support would be very useful and not all that difficult at this point.

This issue is meant to kick off discussion of how this will work, but I have some thoughts as outlined below.

Conventions

In general, we should continue to support native mono y.shape = (N,) without artificial up-casting to explicit mono (1, N).

For example, an stft on (N,) will still produce an output of shape (# freqs, # frames).

However, an stft on (1, N) would produce output (1, #freqs, #frames). Now that framing is fully generalized to multichannel, this should not present any difficulties. More generally, (K, N) would map to (K, #freqs, #frames).

As a general rule, the trailing dimension will be (usually) treated as time-like (samples, frames, etc), and the leading dimension will be channels.

How it will work

Things that will generalize easily

• STFT and friends
• Linear filters (mel, mfcc, chroma_stft, and co)
• HPSS

Things that will be a pain to generalize

• CQT and friends?
• Structure / recurrence
• General decomposition
• effects?
• inverse transforms (mfcc, mel, etc)

Things that should stay monophonic

• Detectors (onset, beat, pitch)
• Display
• Sequence modeling (dtw, viterbi)

[bmcfee]

by works I mean that it produces the correct values in the correct shape. However, the memory layout may or may not adhere to any contiguity constraints. Einsum does claim to let you force this with the order= parameter, but a few minutes of hacking didn’t pay off for me yet.

Update: this should now be fixed thanks to https://github.com/numpy/numpy/pull/16446 being merged.

[bmcfee]

Here’s the relevant discussion on stackoverflow explaining why we are seeing the slowdown and how to optimize it: https://stackoverflow.com/questions/17973507/why-is-converting-a-long-2d-list-to-numpy-array-so-slow

I don’t think that’s the issue we’re having though. The inputs here are already arrays (not lists).