Time-frequency (joint) scattering example/API support

It is my understanding that Joint Time-Frequency Scattering is 2D scattering with a spin up & down Morlet wavelet filterbank, time & frequency directions decoupled, applied on 1D scattering coefficients. If that’s accurate, seems everything’s in place to use it given properly configured filter_bank (think L=2 is needed).

It’d help to have this API-supported with guiding docstrings, or at least an example of how to use Scattering1D & Scattering2D to do joint scattering.

[lostanlen]

3D convs may exploit cross-wavelet dependencies

yes!! an exciting research perspective indeed (never has a neural network been trained on JTFS coefficients)

Multi-channel inputs: it’s preferable to reserve input channels for “channels” rather than flattening them

You’re right. I used the word “channels” in reference to what a typical Conv2D layer would take; but in our case, it’s important to separate the JTFS variable dimension (i.e., containing neither time nor frequency but rather the second-order scales and spin) from the audio channel dimension (e…g, left vs right in stereo music). In full generality, we could have a 5-D output

• time
• frequency
• audio channel (left / right)
• scattering path (scale / rate / spin)
• batch

(i forgot in which order though, so don’t quote me on this)

[cyrusvahidi]

Great to see this is happening and thanks for making this branch available. I have been wondering why Kymatio does not yet support TF scattering.

I am very interested in using this in my research over MATLAB, since it is easily integrated with torch

Will post updates on usage in a pytorch front-end