Move static test data to a submodule

The test vectors significantly increase the repository size (See #240). I personally don’t see the benefit of designing tests against static MATLAB generated test vectors. Many scientific python users now are not using MATLAB anymore and therefore identical bit-exact output is not that crucial anymore. If you really want full MATLAB compatibility, dynamically generated vectors could better cover more test cases.

What about splitting the existing unit tests into functional tests (pure python) and MATLAB compatibility tests which could use oct2py (in case the matlab code runs on octave) or matlab_wrapper to run the tests without the intermediate vectors if available on local machines? For travis obviously only oct2py would work.

[bmcfee]

This issue is back on my mind since we’re doing more intensive github action workflows now. This necessitates pushing around copies of the repo, which can get a bit costly and time-consuming.

To summarize: the proposal — from ahem 5 years ago — is to split tests/data off into a submodule and do a history-rewrite to remove purge the static files as if they were sensitive data.

We would need to also amend the CI action to pull in the submodule, and the contributor guidelines to include instructions on how to run tests.

[bmcfee]

I know I’m warming this up but why don’t we simply leave the static test vectors and put the data into a separate repo while linking back to this repo as a submodule?

Mostly because it’s a lot of work and not a high priority item for me right now. 😁

Seriously though, test refactoring is a huge job that needs to be done ( #391 ), so I’m open to splitting off numerical regression tests as part of that.