testing PoseNet on local files - 'pixels must be HTMLCanvasElement' error

To get help from the community, check out our Google group.

TensorFlow.js version

v0.12.0

Browser version

Node version 10.8.0

Describe the problem or feature request

I’m pretty new to node, so I apologize if this is a simple fix … interested to see how PoseNet could be converted to python. Anyway, I’m trying to run estimateSinglePose on a local file, but I keep getting the following error:

Error: When running in node, pixels must be an HTMLCanvasElement like the one returned by the `canvas` npm package
    at NodeJSKernelBackend.fromPixels (/Users/home/Documents/pose_estimation/poseNet/node_modules/@tensorflow/tfjs-node/dist/nodejs_kernel_backend.js:886:19)
    at Engine.fromPixels (/Users/home/Documents/pose_estimation/poseNet/node_modules/@tensorflow/tfjs-core/dist/engine.js:343:29)
    at fromPixels_ (/Users/home/Documents/pose_estimation/poseNet/node_modules/@tensorflow/tfjs-core/dist/ops/array_ops.js:174:37)
    at Object.fromPixels (/Users/home/Documents/pose_estimation/poseNet/node_modules/@tensorflow/tfjs-core/dist/ops/operation.js:23:29)
    at Object.<anonymous> (/Users/home/Documents/pose_estimation/poseNet/test_git_issue.js:24:21)
    at Module._compile (internal/modules/cjs/loader.js:689:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
    at Module.load (internal/modules/cjs/loader.js:599:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
    at Function.Module._load (internal/modules/cjs/loader.js:530:3)

I’ve tried creating a new canvas Image object with the local file path, but that still produced the error. I also tried passing that image object to a tensor and then sending the tensor to estimateSinglePose, as recommended in this thread, but I keep getting the same error.

Thanks!!

Code to reproduce the bug / link to feature request

const tf = require('@tensorflow/tfjs');
require('@tensorflow/tfjs-node');

const posenet = require('@tensorflow-models/posenet');

const { Image } = require('canvas');

async function singlePoseOutput(imageElement) {
  const imageScaleFactor = 0.50;
  const flipHorizontal = false;
  const outputStride = 16;

  const net = await posenet.load();

  const pose = await net.estimateSinglePose(imageElement, imageScaleFactor, flipHorizontal, outputStride);
  return pose;
};

//create canvas image with local file
const img = new Image();
img.src = './img_file.png';

// convert canvas image to tensor
var img_tensor = tf.fromPixels(img);

const pose = singlePoseOutput(img_tensor);

console.log(pose);

[dsmilkov]

Since you are using the ‘canvas’ npm package, you have to pass the canvas object to fromPixels since that’s the one that polyfills browser’s canvas. The workflow is as follows:

const {createCanvas, Image} = require('canvas');
const canvas = createCanvas(width, height);
const ctx = canvas.getContext('2d');
const img = new Image();
img.src = './img_file.png';
img.onload = () => {
  ctx.drawImage(img, 0, 0);
  const input = tf.fromPixels(canvas);
};

Hope this helps!

[oveddan]

@laurencamblackburn I’m not sure this will solve your problem (it may be node specific), but have you tried making sure the image is loaded first before calling tf.fromPixels?

Check out how the coco demo does it here:

https://github.com/tensorflow/tfjs-models/blob/master/posenet/demos/coco.js#L90

async function loadImage(imageUrl) {
  const image = new Image();
  const promise = new Promise((resolve, reject) => {
    image.crossOrigin = '';
    image.onload = () => {
      resolve(image);
    };
  });

  image.src = imageUrl;
  return promise;
}

const image = await loadImage(imageUrl);

// Creates a tensor from an image
const input = tf.fromPixels(image);