`manage.py --parallel` flag has no effect under django-nose

Django 1.9 introduced the --parallel flag, to run tests in parallel. This gives a 4x speedup on my 4-core Macbook Pro under the default unittest runner, which is pretty great.

Unfortunately I get no effect when I use this flag with the nose test runner.

It does look like the setup is being performed:

$ ./manage.py test --parallel 4
nosetests --logging-clear-handlers --verbosity=1
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Destroying old test database for alias 'default'...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
Cloning test database for alias 'default'...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
Cloning test database for alias 'default'...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
Cloning test database for alias 'default'...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 743 tests in 68.034s

OK
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...

But the test duration is no better than without the --parallel flag.

Is this flag expected to work? I don’t see any documentation for it, though it is explicitly passed through to manage.py here https://github.com/django-nose/django-nose/blob/3b9dad77d0440cace471aa43d77a4ba619f145bb/django_nose/runner.py#L91

[bqumsiyeh]

+1 I’m hitting the same issue as well. Any one have any luck with fixing this?

[Ashish-Bansal]

After adding up configuration, fixing some tests to make it completely work in nose, I found running test is taking 2x time than django’s default runner, and that’s when I found this issue 😦

I think this issue must be mentioned in the docs.

Anyway, I’m not here just to whine. I spent some time to figure out the reason. So, this issue won’t have any workaround other than proper fix in the django-nose code. We can’t use processes because of the reason mentioned by @paultiplady and also all the processes would be sharing the same DB which would cause migrations/fixtures issue.

I don’t have time to fix it up, I’ll probably be switching back to django’s default runner. Here’s some implementation detail in case anyone wants to fix it up:

The way django’s default runner handles this problem is, it partitions the whole testsuite into sub-suites[0] and abstract those multiple suites by subclassing unittest.TestSuite, naming it as ParallelTestSuite[1] . After that, it passes that suite to unittest.TextTestRunner which calls the run method of the suite. In that run[2] method, it creates pool of processes which when initialised, changes the db connections[3] and runs the tests. So, all the processes work upon different db.

To fix this issue in django-nose, the key idea is one will have to do similar stuff of creating multiple processes here[4], run all the sub-suites, keep accumulating results and return it.

[0] https://github.com/django/django/blob/d3449faaa915a08c275b35de01e66a7ef6bdb2dc/django/test/runner.py#L676

[1] https://github.com/django/django/blob/d3449faaa915a08c275b35de01e66a7ef6bdb2dc/django/test/runner.py#L313

[2] https://github.com/django/django/blob/d3449faaa915a08c275b35de01e66a7ef6bdb2dc/django/test/runner.py#L340

[3] https://github.com/django/django/blob/d3449faaa915a08c275b35de01e66a7ef6bdb2dc/django/test/runner.py#L275

[4] https://github.com/django-nose/django-nose/blob/347a711934688feb1d04cd4b17f8aafba995b241/django_nose/runner.py#L244