LSTM examples?

Following on the other issue I created #108 , I’m trying to teach an LSTM network to write a simple children’s book. I’m getting odd behavior but really don’t know what I’m doing to begin with. I’d love to get this example working and added to the readme for others to follow but am hitting lots of little roadblocks. Here’s my code:

const brain = require('brain.js');

const words = new Map();
words.set('00001', 'Jane');
words.set('00010', 'Spot');
words.set('00100', 'Doug');
words.set('01000', 'saw');
words.set('10000', '.');

const trainingData = [
  {input: [0,0,0,0,1], output: [0,1,0,0,0]},// Jane -> saw
  {input: [0,0,0,1,0], output: [0,1,0,0,0]},// Spot -> saw
  {input: [0,0,1,0,0], output: [0,1,0,0,0]},// Doug -> saw
  {input: [0,1,0,0,0], output: [0,0,1,0,0]},// saw -> Doug
  {input: [0,1,0,0,0], output: [0,0,0,1,0]},// saw -> Spot
  {input: [0,1,0,0,0], output: [0,0,0,0,1]},// saw -> Jane
  {input: [0,0,0,1,0], output: [1,0,0,0,0]},// Spot -> .
  {input: [0,0,0,0,1], output: [1,0,0,0,0]},// Jane -> .
  {input: [0,0,1,0,0], output: [1,0,0,0,0]},// Doug -> .
  {input: [1,0,0,0,0], output: [0,0,0,0,1]},// . -> Jane
  {input: [1,0,0,0,0], output: [0,0,0,1,0]},// . -> Spot
  {input: [1,0,0,0,0], output: [0,0,1,0,0]},// . -> Doug
];

const lstm = new brain.recurrent.LSTM();
const result = lstm.train(trainingData);
const run1 = lstm.run([0,0,0,0,1]);// Jane
const run2 = lstm.run(run1);
const run3 = lstm.run(run2);
const run4 = lstm.run(run3);

console.log(words.get('00001'));// start with 'Jane'
console.log(words.get(run1));// saw
console.log(words.get(run2));// Jane, Doug or Spot
console.log(words.get(run3));// .
console.log(words.get(run4));// Jane, Doug or Spot

console.log('run 1:', run1, typeof run1);
console.log('run 2:', run2, typeof run2);
console.log('run 3:', run3, typeof run3);
console.log('run 4:', run4, typeof run4);

The results in the console:

Jane
.
Jane
.
Jane
run 1: 10000 string
run 2: 00001 string
run 3: 10000 string
run 4: 00001 string

some observations:

• the output of run is a string. I was expecting an array

It obviously isn’t working as I expected. Since I can’t find a good example I really have no idea what I’m doing wrong. I’m wondering if I’m training it wrong? I’m starting to think I should be giving it example sequences as input… like {input: "Jane saw Spot", output: "."} but I can’t wrap my head around how to express that as valid input.

[robertleeplummerjr]

TL;DR Let me see if I can get a demo together. Can you supply me with more input data? The recurrent neural networks as they exist aren’t very good with numbers, they are better at translation.

TS;DR The existing recurrent neural network operates like pushing buttons (inputs).

Let that sink in for a minute.

The values that you are training for aren’t actually sent into the neural net (I want to change this). The buttons are defined by training, and is done “by value”. So if you have an input value of say “1.025876” that “value” (singular) isn’t directly sent into the net but rather the buttons that represent the “values” (plural) of “1”, “.”, “0”, “2”, “5”, “8”, “7”, & “6” are ran. That is why LSTM networks are useful in translation, because their inputs are distinct concepts. However, there are many variations that skirt this issue, and it is something that I want to do as well, because in the end, this is all just math. There are many variations of recurrent neural networks, and I’d love to see it implemented so you can just feed it data directly.

Part of my frustration with javascript neural networks is that they are “just hobby architectures”, for learning… As in “don’t take it seriously”… Please.

When I implemented the recurrent neural network from recurrent.js, I took a step back and thought “how can I fix this for javascript?”, for it to not be a hobby, but for it to have that awakening much like when a super hero realizes its powers.

I did a lot of research and arrived at http://gpu.rocks/, but it was not very stable, and not yet released as v1. If you want things done right…

After some initial pull requests, and fantastic conversations with the architects, I was added to the team, where I helped improve the architecture and enabled it to run on node with CPU only fallback, while I was working on brain.js to have an architecture that could use gpu.js. Our plan is to have opencl running in it soon, which is where we go from hobby to enterprise.

Long story short, a few days before the new year, v1 was released. The architecture for FeedForward is far past planning and is being built and tested. I would like to get the library near 100% test coverage, and am taking small steps to achieve that.

The Recurrent neural network concept is being planned, and will match very closely to the feedforward neural network and will allow for this easy means of altering networks to time series data or translation or math or audio or video, because the architecture is all about resembling very simple math.

So getting back to why this response is so long, I could have started on experimenting with more interesting concepts on how to use LSTM’s, and recurrent neural networks, but I chose to build the foundation for making javascript a superhero.

I talked with a fellow developer yesterday and I said the following: “In javascript if you want some serious neural networks to train, it could take a year of training. But if you had the right foundation, you could raise the performance a couple orders of magnitude and calculate those same values in a day. I’d take a year to build that foundation. Why? Because then I could do 4 lifetimes of work in two years.”

[robertleeplummerjr]

The underlying engine (GPU.js) is in very active development, has been proven to work in node and will be released soon, which will then spark the completion of v2 of brain.js, including the lstm bits.