Fix source separation unit tests

Tagging @faroit @ecmjohnson

At some point in the last six months, the regression tests for source separation have started failing, specifically on fixtures output00 (Images Frames - Source to Interference and Artifact) and output04 (Images - Source to Interference and Artifact).

Anyone know what’s going on here? The results are off by up to 1e-2 in some cases, so I don’t think this is up to json encoding round-off error. Maybe something changed under the hood in numpy or scipy since the fixtures were generated? If that’s all it is, then it should be a simple matter of revising the expected scores.

It looks like these fixtures were most recently updated by @ecmjohnson in #212 .

[craffel]

B) relax the tolerance for passing the tests.

This seems fine to me.

[bmcfee]

I guess dropping blas-enabled numpy builds for the unit tests is not an option for you anaconda users, right?

They’re all blas enabled, it’s just a question of which blas. We could try to do a better job of matching the outputs to the test environments, but I think that’s sort of missing the point, and wouldn’t ensure that different users would get the same results on the same input anyway.

did you test with stock numpy vs. mkl or did you try even more blas backends?

I tried locally on these two numpy builds.

This one passes:

In [6]: np.__config__.show()
lapack_opt_info:
    library_dirs = ['/home/bmcfee/miniconda/envs/clean/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/bmcfee/miniconda/envs/clean/include']
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']
blas_opt_info:
    library_dirs = ['/home/bmcfee/miniconda/envs/clean/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/bmcfee/miniconda/envs/clean/include']
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']
lapack_mkl_info:
    library_dirs = ['/home/bmcfee/miniconda/envs/clean/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/bmcfee/miniconda/envs/clean/include']
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']
openblas_lapack_info:
  NOT AVAILABLE
blas_mkl_info:
    library_dirs = ['/home/bmcfee/miniconda/envs/clean/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/home/bmcfee/miniconda/envs/clean/include']
    libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'iomp5', 'pthread']

This one fails:

In [2]: np.__config__.show()
lapack_mkl_info:
  NOT AVAILABLE
openblas_info:
    language = c
    library_dirs = ['/home/bmcfee/miniconda/envs/py35/lib']
    define_macros = [('HAVE_CBLAS', None)]
    libraries = ['openblas', 'openblas']
blas_opt_info:
    language = c
    library_dirs = ['/home/bmcfee/miniconda/envs/py35/lib']
    define_macros = [('HAVE_CBLAS', None)]
    libraries = ['openblas', 'openblas']
blis_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
    language = c
    library_dirs = ['/home/bmcfee/miniconda/envs/py35/lib']
    define_macros = [('HAVE_CBLAS', None)]
    libraries = ['openblas', 'openblas']
lapack_opt_info:
    language = c
    library_dirs = ['/home/bmcfee/miniconda/envs/py35/lib']
    define_macros = [('HAVE_CBLAS', None)]
    libraries = ['openblas', 'openblas']