Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Memory Leak with GPU & tf.node.decodeImage

See original GitHub issue

Windows 10 20H2, Cuda 10.0, Cudnn v7.6.5, TFJS v2.7.0

Issue Running the following code fills my 1080s memory causing issues when performing any actual useful code

const { node: { decodeImage }, tidy, engine } = require('@tensorflow/tfjs-node-gpu')
const fs = require('fs')

async function main () {
  engine().startScope()
  const buffer = fs.readFileSync('./test.jpg')
  const tensor = tidy(() => decodeImage(Buffer.from(buffer)).toFloat().expandDims())
  tensor.dispose()
  engine().endScope()
  setTimeout(() => {
  }, 5000)
}
main()

I tried without scopes & tidy. Same result. GPU usage goes back to normal after the timeout ends & the script exists.

Issue Analytics

State:
Created 3 years ago
Reactions:5
Comments:15 (1 by maintainers)

Top GitHub Comments

4reactions

WilledgeRcommented, Mar 26, 2021

same issue on tf-js-node cpu

2reactions

sudo-carsoncommented, Apr 17, 2021

I’m having the same problem with the tfjs-node CPU backend, but only for certain JPEGs shot on iPhones. Some cause a memory leak, some don’t.

If it helps, here’s the header of a leaky image:

FF D8 FF E1 00 22 45 78 69 66 00 00 4D 4D 00 2A 00 00 00 08 00 01 01 12 00 03 00 00 00 01 00 01 00 00 00 00 00 00 FF ED 00 38 50 68 6F 74 6F 73 68 6F 70 20 33 2E 30 00 38 42 49 4D 04 04 00 00 00 00 00 00 38 42 49 4D 04 25 00 00 00 00 00 10 D4 1D 8C D9 8F 00 B2 04 E9 80 09 98 EC F8 42 7E FF E2 02 34 49 43 43 5F 50 52 4F 46 49 4C 45 00 01 01 00 00 02 24 61 70 70 6C 04 00 00 00 6D 6E 74 72 52 47 42 20 58 59 5A 20 07 E1 00 07 00 07 00 0D 00 16 00 20 61 63 73 70 41 50 50 4C 00 00 00 00 41 50 50 4C 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 F6 D6 00 01 00 00 00 00 D3 2D 61 70 70 6C CA 1A 95 82 25 7F 10 4D 38 99 13 D5 D1 EA 15 82 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0A 64 65 73 63 00 00 00 FC 00 00 00 65 63 70 72 74 00 00 01 64 00 00 00 23 77 74 70 74 00 00 01 88 00 00 00 14 72 58 59 5A 00 00 01 9C 00 00 00 14 67 58 59 5A 00 00 01 B0 00 00 00 14 62 58 59 5A 00 00 01 C4 00 00 00 14 72 54 52 43 00 00 01 D8 00 00 00 20 63 68 61 64 00 00 01 F8 00 00 00 2C 62 54 52 43 00 00 01 D8 00 00 00 20 67 54 52 43 00 00 01 D8 00 00 00 20 64 65 73 63 00 00 00 00 00 00 00 0B 44 69 73 70 6C 61 79 20 50 33 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 78 74 00 00 00 00 43 6F 70 79 72 69 67 68 74 20 41 70 70 6C 65 20 49 6E 63 2E 2C 20 32 30 31 37 00 00 58 59 5A 20 00 00 00 00 00 00 F3 51 00 01 00 00 00 01 16 CC 58 59 5A 20 00 00 00 00 00 00 83 DF 00 00 3D BF FF FF FF BB 58 59 5A 20 00 00 00 00 00 00 4A BF 00 00 B1 37 00 00 0A B9 58 59 5A 20 00 00 00 00 00 00 28 38 00 00 11 0B 00 00 C8 B9 70 61 72 61 00 00 00 00 00 03 00 00 00 02 66 66 00 00 F2 A7 00 00 0D 59 00 00 13 D0 00 00 0A 5B 73 66 33 32 00 00 00 00 00 01 0C 42 00 00 05 DE FF FF F3 26 00 00 07 93 00 00 FD 90 FF FF FB A2 FF FF FD A3 00 00 03 DC 00 00 C0 6E FF C0 00 11 08 0F C0 0B D0 03 01 22 00 02 11 01 03 11 01 FF C4 00 1F 00 00 01 05 01 01 01 01 01 01 00 00 00 00 00 00 00 00 01 02 03 04 05 06 07 08 09 0A 0B FF C4 00 B5 10 00 02 01 03 03 02 04 03 05 05 04 04 00 00 01 7D 01 02 03 00 04 11 05 12 21 31 41 06 13 51 61 07 22 71 14 32 81 91 A1 08 23 42 B1 C1 15 52 D1 F0 24 33 62 72 82 09 0A 16 17 18 19 1A 25 26 27 28 29 2A 34 35 36 37 38 39 3A 43 44 45 46 47 48 49 4A 53 54 55 56 57 58 59 5A 63 64 65 66 67 68 69 6A 73 74 75 76 77 78 79 7A 83 84 85 86 87 88 89 8A 92 93 94 95 96 97 98 99 9A A2 A3 A4 A5 A6 A7 A8 A9 AA B2 B3 B4 B5 B6 B7 B8 B9 BA C2 C3 C4 C5 C6 C7 C8 C9 CA D2 D3 D4 D5 D6 D7 D8 D9 DA E1 E2 E3 E4 E5 E6 E7 E8 E9 EA F1 F2 F3 F4 F5 F6 F7 F8 F9 FA FF C4 00 1F 01 00 03 01 01 01 01 01 01 01 01 01 00 00 00 00 00 00 01 02 03 04 05 06 07 08 09 0A 0B FF C4 00 B5 11 00 02 01 02 04 04 03 04 07 05 04 04 00 01 02 77 00 01 02 03 11 04 05 21 31 06 12 41 51 07 61 71 13 22 32 81 08 14 42 91 A1 B1 C1 09 23 33 52 F0 15 62 72 D1 0A 16 24 34 E1 25 F1 17 18 19 1A 26 27 28 29 2A 35 36 37 38 39 3A 43 44 45 46 47 48 49 4A 53 54 55 56 57 58 59 5A 63 64 65 66 67 68 69 6A 73 74 75 76 77 78 79 7A 82 83 84 85 86 87 88 89 8A 92 93 94 95 96 97 98 99 9A A2 A3 A4 A5 A6 A7 A8 A9 AA B2 B3 B4 B5 B6 B7 B8 B9 BA C2 C3 C4 C5 C6 C7 C8 C9 CA D2 D3 D4 D5 D6 D7 D8 D9 DA E2 E3 E4 E5 E6 E7 E8 E9 EA F2 F3 F4 F5 F6 F7 F8 F9 FA FF DB 00 43 00 04 04 04 04 04 04 06 04 04 06 09 06 06 06 09 0C 09 09 09 09 0C 0F 0C 0C 0C 0C 0C 0F 12 0F 0F 0F 0F 0F 0F 12 12 12 12 12 12 12 12 15 15 15 15 15 15 19 19 19 19 19 1C 1C 1C 1C 1C 1C 1C 1C 1C 1C FF DB 00 43 01 04 05 05 07 07 07 0C 07 07 0C 1D 14 10 14 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D 1D FF DD 00 04 00 BD

And the header of a non-leaky image:

FF D8 FF E1 00 16 45 78 69 66 00 00 4D 4D 00 2A 00 00 00 08 00 00 00 00 00 00 FF E2 02 34 49 43 43 5F 50 52 4F 46 49 4C 45 00 01 01 00 00 02 24 61 70 70 6C 04 00 00 00 6D 6E 74 72 52 47 42 20 58 59 5A 20 07 E1 00 07 00 07 00 0D 00 16 00 20 61 63 73 70 41 50 50 4C 00 00 00 00 41 50 50 4C 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 F6 D6 00 01 00 00 00 00 D3 2D 61 70 70 6C CA 1A 95 82 25 7F 10 4D 38 99 13 D5 D1 EA 15 82 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0A 64 65 73 63 00 00 00 FC 00 00 00 65 63 70 72 74 00 00 01 64 00 00 00 23 77 74 70 74 00 00 01 88 00 00 00 14 72 58 59 5A 00 00 01 9C 00 00 00 14 67 58 59 5A 00 00 01 B0 00 00 00 14 62 58 59 5A 00 00 01 C4 00 00 00 14 72 54 52 43 00 00 01 D8 00 00 00 20 63 68 61 64 00 00 01 F8 00 00 00 2C 62 54 52 43 00 00 01 D8 00 00 00 20 67 54 52 43 00 00 01 D8 00 00 00 20 64 65 73 63 00 00 00 00 00 00 00 0B 44 69 73 70 6C 61 79 20 50 33 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 74 65 78 74 00 00 00 00 43 6F 70 79 72 69 67 68 74 20 41 70 70 6C 65 20 49 6E 63 2E 2C 20 32 30 31 37 00 00 58 59 5A 20 00 00 00 00 00 00 F3 51 00 01 00 00 00 01 16 CC 58 59 5A 20 00 00 00 00 00 00 83 DF 00 00 3D BF FF FF FF BB 58 59 5A 20 00 00 00 00 00 00 4A BF 00 00 B1 37 00 00 0A B9 58 59 5A 20 00 00 00 00 00 00 28 38 00 00 11 0B 00 00 C8 B9 70 61 72 61 00 00 00 00 00 03 00 00 00 02 66 66 00 00 F2 A7 00 00 0D 59 00 00 13 D0 00 00 0A 5B 73 66 33 32 00 00 00 00 00 01 0C 42 00 00 05 DE FF FF F3 26 00 00 07 93 00 00 FD 90 FF FF FB A2 FF FF FD A3 00 00 03 DC 00 00 C0 6E FF DB 00 84 00 09 06 07 10 10 10 1F 10 10 10 15 1A 20 20 10 0F 17 10 10 10 1E 28 20 16 1A 20 0D 28 1E 20 28 28 28 1A 28 2F 31 20 30 20 29 31 0E 1E 25 38 25 26 2E 40 41 2B 2C 28 25 2C 2E 3C 30 34 29 27 41 21 01 0A 0A 0A 0F 10 0D 0D 0D 15 11 2B 0D 15 15 29 2D 31 3B 2A 36 31 2D 33 29 33 34 25 37 3A 2D 35 32 33 2A 25 21 2B 29 31 30 36 2C 33 36 32 2F 36 37 2C 31 26 33 2C 33 25 2C 31 31 2E 26 2F 30 24 39 FF C0 00 11 08 03 6C 03 6C 03 01 22 00 02 11 01 03 11 01 FF C4 00 1C 00 00 03 01 01 01 01 01 01 00 00 00 00 00 00 00 00 00 01 02 03 04 05 06 07 08 FF C4 00 3C 10 00 02 02 01 03 02 04 05 02 05 04 01 04 02 03 01 00 01 02 11 03 04 21 31 12 41 05 51 61 81 13 22 71 91 A1 32 B1 06 14 42 C1 F0 52 D1 E1 F1 62 72 82 92 C2 23 33 15 43 A2 16 FF C4 00 18 01 01 01 01 01 01 00 00 00 00 00 00 00 00 00 00 00 00 01 02 03 04 FF C4 00 1D 11 01 01 01 01 00 03 01 01 01 00 00 00 00 00 00 00 00 01 11 02 12 21 31 41 03 61

Seems like it could have something to do with Exif data?