[tfjs-models] posenet pngs 'Incompatible shapes'

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js): yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): arch linux Linux 5.11.12-zen1-1-zen x86_64
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: no
• TensorFlow.js installed from (npm or script link): https://www.npmjs.com/package/@tensorflow-models/posenet
• TensorFlow.js version (use command below): posenet: 2.2.2, tfjs-core: 3.4.0, tfjs-node-gpu: 3.4.0
• Browser version: n/a (using nodejs v15.14.0)
• Tensorflow.js Converter Version: 3.4.0

Describe the current behavior I script I have attached below throws an exception when some (but not all) pngs are passed to it. The error is:

Error: Invalid TF_Status: 3
Message: Incompatible shapes: [193,257,4] vs. [3]

Describe the expected behavior tfjs should be able to decode any png in a manner that is acceptable by posenet.

Standalone code to reproduce the issue

import '@tensorflow/tfjs-node-gpu'
import fs from 'fs'
import * as posenet from '@tensorflow-models/posenet'
import * as tf from '@tensorflow/tfjs-node-gpu'

const resnet_portrait_config: posenet.ModelConfig = {
  architecture: 'ResNet50',
  outputStride: 32,
  inputResolution: { width: 200, height: 257 },
  quantBytes: 2,
}

;(async () => {
  const image_buffer = await fs.promises.readFile('out1.jpg')
  const channels = 0
  const input = tf.node.decodePng(image_buffer, channels)
  const net = await posenet.load(resnet_portrait_config)
  const pose = await net.estimateSinglePose(input, {
    flipHorizontal: false,
  })
})()

Other info / logs I dont fully understand what defines a Tensor3D vs Tensor4D, but I am guessing this error is because decodePng is generating a Tensor4D, which posenet does not know what to do with. Its possible the typings are outdated, they say that decodePng should only return Tensor3D, but they also say that input.rankType should equal 'R3', when it actually is '3'. Here is the tensor printed out:

Tensor {
  kept: false,
  isDisposedInternal: false,
  shape: [ 1114, 934, 4 ],
  dtype: 'int32',
  size: 4161904,
  strides: [ 3736, 4 ],
  dataId: {},
  id: 2,
  rankType: '3',
  scopeId: 0
}

full stacktrace:

An unexpected error occurred:
Error: Invalid TF_Status: 3
Message: Incompatible shapes: [193,257,4] vs. [3]
    at NodeJSKernelBackend.executeSingleOutput (/home/andrew/Code/development/posenet-find-similar-poses/node_modules/.pnpm/@tensorflow+tfjs-node-gpu@3.4.0/node_modules/@tensorflow/tfjs-node-gpu/dist/nodejs_kernel_backend.js:209:43)
    at Object.kernelFunc (/home/andrew/Code/development/posenet-find-similar-poses/node_modules/.pnpm/@tensorflow+tfjs-node-gpu@3.4.0/node_modules/@tensorflow/tfjs-node-gpu/dist/kernels/Add.js:28:24)
    at kernelFunc (/home/andrew/Code/development/posenet-find-similar-poses/node_modules/.pnpm/@tensorflow+tfjs-core@3.4.0/node_modules/@tensorflow/tfjs-core/src/engine.ts:636:22)
    at /home/andrew/Code/development/posenet-find-similar-poses/node_modules/.pnpm/@tensorflow+tfjs-core@3.4.0/node_modules/@tensorflow/tfjs-core/src/engine.ts:703:23
    at Engine.scopedRun (/home/andrew/Code/development/posenet-find-similar-poses/node_modules/.pnpm/@tensorflow+tfjs-core@3.4.0/node_modules/@tensorflow/tfjs-core/src/engine.ts:478:19)
    at Engine.runKernelFunc (/home/andrew/Code/development/posenet-find-similar-poses/node_modules/.pnpm/@tensorflow+tfjs-core@3.4.0/node_modules/@tensorflow/tfjs-core/src/engine.ts:699:10)
    at Engine.runKernel (/home/andrew/Code/development/posenet-find-similar-poses/node_modules/.pnpm/@tensorflow+tfjs-core@3.4.0/node_modules/@tensorflow/tfjs-core/src/engine.ts:543:17)
    at add_ (/home/andrew/Code/development/posenet-find-similar-poses/node_modules/.pnpm/@tensorflow+tfjs-core@3.4.0/node_modules/@tensorflow/tfjs-core/src/ops/add.ts:57:17)
    at Object.add__op [as add] (/home/andrew/Code/development/posenet-find-similar-poses/node_modules/.pnpm/@tensorflow+tfjs-core@3.4.0/node_modules/@tensorflow/tfjs-core/src/ops/operation.ts:51:24)
    at ResNet.preprocessInput (/home/andrew/Code/development/posenet-find-similar-poses/node_modules/.pnpm/@tensorflow-models+posenet@2.2.2_471dfd93c59a4b9679f4a8b7b4d0b378/node_modules/@tensorflow-models/posenet/src/resnet.ts:26:15)

if I use imagemagick to convert the image to a jpg, posenet does not throw an error, though the score is extremely low.

[vladmandic]

• 0 means leave number of channels as-is in the input image and it’s default so no need to specify it.
• 1 means reduce to one channel which means black & white image
• 3 means reduce to three channels which means RGB image

no harm done with having 3 as parameter always as if image is already RGB, it will stay RGB and if its RGBA, it will be stripped of alpha channel and return is again RGB.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No