New Feature: Multithreaded Training

(I’m opening a new issue for this to start a conversation before submitting a pull request, so please let me know what you think.)

This adds new functionality to trainAsyc(), so that NodeJS users can utilize multiple gpus and cpus to train a single NeuralNetwork. This should significantly speed up training if you have a large neural net and/or a large training data set.

Is this a feature that we would want to merge into develop? [y/n]

Code

This branch, based on master, has a working example: massaroni/feature-parallel-training-m This other branch is mergeable into develop, but develop is too unstable at this point to demo the multithreaded training. massaroni/feature-parallel-training

See the example in parallel-trainer-example.js. It basically just shows that the algorithm does converge. See the main functionality in parallel-trainer.js

Documentation

trainAsync() in parallel mode, can train a single net on multiple threads. This should speed up training for large nets, large training sets, or both.

Train a NeuralNetwork on 3 cpu threads.

  const net = new brain.NeuralNetwork();
  net
    .trainAsync(data, {
      parallel: {
        threads: 3,
        partitionSize: 1500, // optional. send a partition of 1500 items from the training set to each thread.  Raise this number to get some overlap in the training data partitions.
        epochs: 20000, // optional. limit each thread to 20,000 training runs
      },
      // ... and the usual training options
    })
    .then(res => {
      // do something with my trained network
    })
    .catch(handleError);

Train a NeuralNetwork on 6 cpu threads and 2 GPU threads.

  const net = new brain.NeuralNetwork();
  net
    .trainAsync(data, {
      parallel: {
        threads: {
          NeuralNetwork: 6,
          NeuralNetworkGPU: 2
        }
      },
      // ... and the usual training options
    })
    .then(res => {
      // do something with my trained network
    })
    .catch(handleError);

Train a single NeuralNetwork on 6 cpu threads and 2 GPU threads, and send 10x more training data to the GPUs because they can run through it faster.

  const net = new brain.NeuralNetwork();
  net
    .trainAsync(data, {
      parallel: {
        threads: {
          NeuralNetwork: {
            threads: 6,
            trainingDataSize: 2200
          },
          NeuralNetworkGPU: {
            threads: 2,
            trainingDataSize: 22000
          }
        }
      },
      // ... and the usual training options
    })
    .then(res => {
      // do something with my trained network
    })
    .catch(handleError);

Roadmap

• support all other neural net types
• web workers, for multithreaded training in the browser
• distributed training (multiple machines) (async SGD w/stale gradient handling?)

[blackforest-t]

I would like to donate something if I could use four cores at the same time instead of one, in my case! Is this also intended for LSTM (not LSTMTimeStep)?

i’ll like to know about it too

[Rocketblaster247]

Anything to make training faster!