TypeError: Cannot read property 'backend' of undefined - tensor is already disposed when moveData is called.

I’m trying to setup an LSTM, but Tensorflow.js doesn’t seem to like it very much. It keeps crashing with the following error:

Epoch 1 / 50
eta=0.0 --------------------------------------------------------- categoricalCrossentropy=1.04e-5 loss=1.04e-5 /home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3280
        var srcBackend = info.backend;
                              ^

TypeError: Cannot read property 'backend' of undefined
    at Engine.moveData (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3280:31)
    at DataStorage.get (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:115:28)
    at NodeJSKernelBackend.getInputTensorIds (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:153:43)
    at NodeJSKernelBackend.executeSingleOutput (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:200:73)
    at NodeJSKernelBackend.reshape (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:1055:21)
    at forward (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:7430:24)
    at /home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3480:55
    at /home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3319:22
    at Engine.scopedRun (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3329:23)
    at Engine.tidy (/home/sbrl/Documents/code/javascript/node/byte-lstm/node_modules/@tensorflow/tfjs-core/dist/tf-core.node.js:3318:21)

Is there any reason for this? I’ve double-checked all my tensors going in, and they all contain valid data.

System details:

• Intel® Core™ i7-10875H CPU @ 2.30GHz
• Ubuntu 20.10
• Nvidia GeForce RTX 2060 Driver Version: 455.38, CUDA Version: 10.0 (11.1 installed globally, 10.0 installed locally, error occurs regardless of whether I use the GPU or not)
• Packages and versions: @tensorflow/tfjs-node = 2.7.0, @tensorflow/tfjs-node-gpu = 2.7.0

Model summary:

_________________________________________________________________
Layer (type)                 Output shape              Param #   
=================================================================
lstm_LSTM1 (LSTM)            [64,32,1]                 12        
_________________________________________________________________
lstm_LSTM2 (LSTM)            [64,32,32]                4352      
_________________________________________________________________
lstm_LSTM3 (LSTM)            [64,32,1]                 136       
=================================================================
Total params: 4500
Trainable params: 4500
Non-trainable params: 0
_________________________________________________________________

Options objects for the 3 layers:

{
  stateful: true,
  units: 1,
  returnSequences: true,
  inputShape: [ 32, 1 ],
  batchSize: 64
}
{ stateful: true, units: 32, returnSequences: true }
{ stateful: true, units: 1, returnSequences: true }

• Model creation code: https://ybin.me/p/3b2151c72409317f#IXhSzO+VS1abLA9RXjX4JMtQf/oVbGUFe+xTqLoFcNE=
• Dataset creation code: https://ybin.me/p/22a21ae11489ef2c#eZaLy5MgWwi7L3EFo5D+jHsG8xAv7mp+8suvTPbXHqY=

The dataset is created by passing Uint8Array instances to tf.tensor, which are themselves views of a main ArrayBuffer.

Training code:

await this.model.fitDataset(dataset, {
	epochs: 50,
	verbose: 1,
	yieldEvery: "batch",
	shuffle: false,
	callbacks: {
		onEpochEnd: async (epoch, metrics) => {
			// .....
		}
	}
});

…full code available upon request.

[tafsiri]

You can leave it open for now. We can discuss internally what the best solution might be (there are a number of different places we may want to tackle this). I’m going to update the title for when we get a chance to look more closely at this.

[sbrl]

Oops, I did NOT mean to close this! Reopened. Hopefully that doesn’t affect anything @tafsiri @rthadur?