Framewise separation evaluation where some frames are silent

While evaluating the framewise separation methods that have been added #199 and are almost ready in #212 , I’ve figured out, the validation routines are a bit too strict: Let’s assume there is silence in one of the frames being evaluated:

nsrc = 2
nchan = 2

empty_source = np.zeros((nsrc, 20, nchan))
estimates = np.random.random_sample((nsrc, 200, nchan))

estimated_sources = np.concatenate(
    (empty_source, estimates),
    axis=1
)

mir_eval.separation.bss_eval_images_framewise(
    estimated_sources,
    estimated_sources,
    window=10,
    hop=5
)

this results in

ValueError: All the reference sources should be non-silent (not all-zeros), but at least one of the reference sources is all 0s, which introduces ambiguity to the evaluation. (Otherwise we can add infinitely many all-zero sources.)

and therefore would not continue evaluating the other (non-silent) frames. I think it would make sense to return np.nan or None for those (2) frames in the framewise evaluation. I don’t know how to handle these results later when it comes to compute aggregated measures, but thats more of a discussion for SISEC (@aliutukus ?)

@craffel @ErikJohnsonCU @carlthome what do you think?

[ecmjohnson]

@craffel @craffel @bmcfee @dpwe

It’s been a pleasure being able to contribute and help mir_eval grow a little bit! Thank you so much to @faroit for hosting me; I really enjoyed the experience of working @audiolabs! I’ve learned a lot and I really appreciate all the nitpicks that have helped me refine my python style and understanding of open source software contributing.

[craffel]

Resolved in 8a1cb7210354ebff5d776a034f6df83132c85fe5. Thanks!