Webcam feed on NodeJs environment

Hi,

I would like to perform live face detection in a nodejs environment but I don’t quiet see how to do that.

Initially, I was capturing frames from my webcam using opencv4nodejs:

cap = new cv.VideoCapture(videoDeviceId);
(...)
_frame = cap.read();

But then I would have to find a way to convert _frame to a Canvas, right ? At least if I’m using a monkey path like this:

const { Canvas, Image, ImageData } = canvas
faceapi.env.monkeyPatch({ Canvas, Image, ImageData })

Do I have other options ? Would it be possible to use a monkey patch with some opencv4nodejs’ elements ?

I saw this line in README.md:

Alternatively you can simply construct your own tensors from image data and pass tensors as inputs to the API.

Okay, but how can I achieve that ? If I look at detectAllFaces’ signature, I can see the following:

detectAllFaces(input: TNetInput, options?: FaceDetectionOptions): DetectAllFacesTask

And if I look further, I can see that TNetInput can be a TNetInputArg, which can be a TResolvedNetInput, which can be a tf.Tensor3D or tf.Tensor4D.

So I would have to retrieve data from _frame and use them to instantiate a tf.Tensor3D ?

Thanks !

[mamacker]

In case someone comes along after me looking for exactly the code needed. As a repo: https://github.com/mamacker/faceme

import * as tfjs from '@tensorflow/tfjs-node';
import * as faceapi from 'face-api.js';
import * as fs from 'fs';
import * as cv from 'opencv';

const minConfidence = 0.5;
/*
const faceDetectionNet = faceapi.nets.ssdMobilenetv1;
const faceDetectionOptions = new faceapi.SsdMobilenetv1Options({minConfidence});
*/
const faceDetectionNet = faceapi.nets.tinyFaceDetector;
const faceDetectionOptions = new faceapi.TinyFaceDetectorOptions({minConfidence});

let globalFrame = null;
let lastFrame = null;
function processFrame() {
  if (globalFrame == null) return;
  if (lastFrame == globalFrame) return;

  lastFrame = globalFrame;
  let tFrame = faceapi.tf.tensor3d(globalFrame, [480, 640, 3])
  faceapi.detectAllFaces(tFrame, faceDetectionOptions).then((faces) => {
    console.log(faces);
  });
}
setInterval(processFrame, 100);

async function run() {
  await faceDetectionNet.loadFromDisk('./weights');
  let video = '/dev/video0';
  let cap = new cv.VideoCapture(video);
  let capture = () => {
    cap.read(function(err, frame) {
      if (frame.width() > 0) {
        let data = new Uint8Array(frame.getData().buffer);
        globalFrame = data;
      }

      setTimeout(capture, 0);
    });
  }
  capture();
}

run()

[msvargas]

Thank you @mamacker for the code 🙇‍♂️ Thanks to you I was able to remove canvas from my project 🙇‍♂️

Full code: https://github.com/whyboris/extract-faces-node

I’m using sharp for image loading (as I need other image manipulations later)

Relevant piece:

const sharp = require('sharp');
const imgBuffer: Buffer = await sharp('./some/img/path.jpg').toBuffer();
const imgTensor = tf.node.decodeJpeg(imgBuffer);
const detections = await faceapi.detectAllFaces(imgTensor);

This the correct mode, using decodeJpeg and most faster