Framewise separation evaluation where some frames are silent
See original GitHub issueWhile evaluating the framewise separation methods that have been added #199 and are almost ready in #212 , I’ve figured out, the validation routines are a bit too strict: Let’s assume there is silence in one of the frames being evaluated:
nsrc = 2
nchan = 2
empty_source = np.zeros((nsrc, 20, nchan))
estimates = np.random.random_sample((nsrc, 200, nchan))
estimated_sources = np.concatenate(
(empty_source, estimates),
axis=1
)
mir_eval.separation.bss_eval_images_framewise(
estimated_sources,
estimated_sources,
window=10,
hop=5
)
this results in
ValueError: All the reference sources should be non-silent (not all-zeros), but at least one of the reference sources is all 0s, which introduces ambiguity to the evaluation. (Otherwise we can add infinitely many all-zero sources.)
and therefore would not continue evaluating the other (non-silent) frames. I think it would make sense to return np.nan
or None
for those (2) frames in the framewise evaluation. I don’t know how to handle these results later when it comes to compute aggregated measures, but thats more of a discussion for SISEC (@aliutukus ?)
@craffel @ErikJohnsonCU @carlthome what do you think?
Issue Analytics
- State:
- Created 7 years ago
- Comments:10 (10 by maintainers)
@craffel @craffel @bmcfee @dpwe
It’s been a pleasure being able to contribute and help
mir_eval
grow a little bit! Thank you so much to @faroit for hosting me; I really enjoyed the experience of working @audiolabs! I’ve learned a lot and I really appreciate all the nitpicks that have helped me refine my python style and understanding of open source software contributing.Resolved in 8a1cb7210354ebff5d776a034f6df83132c85fe5. Thanks!