Improvements for scanning: choose best resolution and crop the area that should be checked

First of all, thank you for all the efforts you put into this OS project. This should not be taken for granted!

We use the ngx-scanner in an angular application to scan various different types of codes in different sizes from different materials (printed paper, laser marked metal parts, …). Especially the laser marked codes are tough because the codes are not super accurate, small and the metal surface might have scratches and reflect the light. So we spend quite some time improving the scanning with the ngx-scanner that maybe others would also be interested in.

What we have customized so far:

• Use the highest camera resolution possible
• Define / Crop the area to search for codes
• Support for inverted codes (white code on black material)

We did the tweaks by overriding methods on the BrowserCodeReader and ZXingScannerComponent. Especially the first two adjustments might be something that could be considered to add as feature into the component/library directly.

Use the highest camera resolution possible

Is your feature request related to a problem? Please describe. By default it always choose a

• 640x480 resolution for my laptop webcam in chrome (even tough it would support 1280x720)
• 640x480 resolution for the IPad in safari (even tough it would support 1920x1080)

After some research we stumbled over the constraints (https://developer.mozilla.org/en-US/docs/Web/API/Media_Streams_API/Constraints) and tweaked the two places where the constraints where applied:

ZXingScannerComponent.prototype.getAnyVideoDevice = (): Promise<MediaStream> => {
  return navigator.mediaDevices.getUserMedia({
    audio: false,
    video: {
      width: { min: 640, ideal: 1920 },
      height: { min: 400, ideal: 1080 },
      aspectRatio: { ideal: 1.7777777778 }
    }
  });
};

BrowserCodeReader.prototype.decodeFromVideoDevice = async function(
  deviceId: string | undefined,
  previewElem: string | HTMLVideoElement | undefined,
  callbackFn: any,
): Promise<IScannerControls> {

  // We had to comment this out because it is a private method...
  // BrowserCodeReader.checkCallbackFnOrThrow(callbackFn);

  let videoConstraints: MediaTrackConstraints;

  if (!deviceId) {
    videoConstraints = { facingMode: 'environment' };
  } else {
    videoConstraints = {
      deviceId: { exact: deviceId },
      width: { min: 640, ideal: 1920 },
      height: { min: 400, ideal: 1080 },
      aspectRatio: { ideal: 1.7777777778 }
    };
  }

  const constraints: MediaStreamConstraints = { video: videoConstraints };

  return await this.decodeFromConstraints(constraints, previewElem, callbackFn);
};

This lead to a higher resolution and therfore better scans.

Describe the solution you’d like I think it might be useful to make the constraints customizable (as input of the component maybe), and also change the default to something like:

{
  video: {
    width: { min: 640, ideal: 1920 },
    height: { min: 400, ideal: 1080 },
    aspectRatio: { ideal: 1.7777777778 }
  }
}

Additional context

There might be a need for some further investigations for other browser, because they all tend to interpret the constraints a little bit different. Here is a nice website that lets you test common constraints for your camera: https://webrtchacks.github.io/WebRTC-Camera-Resolution/

Define / Crop the area to search for codes

Is your feature request related to a problem? Please describe. We added two different modes to our scanner to crop the scanned area for 1D and 2D codes:

This was a bit difficult because of the different resolutions / sizes and orientations of the devices but what we basically do is some calculation where the displayed cropping area will be found in the actual canvas, and fill a new canvas with only the cropped part to evaluate in the code detection:

BrowserCodeReader.createBinaryBitmapFromCanvas = function (canvas: HTMLCanvasElement): BinaryBitmap {

  const cameraCanvasImageWidth = canvas.width;
  const cameraCanvasImageHeight = canvas.height;
  const videoElementHeight = (document.querySelector('zxing-scanner video') as any).offsetHeight;
  const videoElementWidth = (document.querySelector('zxing-scanner video') as any).offsetWidth;

  let factor = 1;
  if (videoElementWidth > videoElementHeight) {
    factor = cameraCanvasImageWidth / videoElementWidth;
  } else {
    factor = cameraCanvasImageHeight / videoElementHeight;
  }

  const width = Math.floor((this.squareShape ? 250 : 450) * factor);
  const height = Math.floor((this.squareShape ? 250 : 200) * factor);

  const left = (cameraCanvasImageWidth - width) / 2;
  const top = (cameraCanvasImageHeight - height) / 2;

  const croppedCanvas = document.createElement('canvas');
  croppedCanvas.width = width;
  croppedCanvas.height = height;
  const croppedCanvasContext = croppedCanvas.getContext('2d');
  croppedCanvasContext.rect(0, 0, width, height);
  croppedCanvasContext.fillStyle = 'white';
  croppedCanvasContext.fill();
  croppedCanvasContext.drawImage(canvas, left, top, width, height, 0, 0, width, height);

  // These lines can be used to show the cropped part of the image stream that is used
  // to find the code and check if the highlighted area matches the cropped image
  // document.getElementById('croppedSvg').innerHTML = '';
  // const span = document.createElement('span');
  // span.textContent = `${cameraCanvasImageWidth} x ${cameraCanvasImageHeight} | ${videoElementWidth} x ${videoElementHeight}`;
  // span.style.position = 'absolute';
  // span.style.right = `0`;
  // span.style.color = 'white';
  // span.style.display = 'block';
  // document.getElementById('croppedSvg').appendChild(span);
  // croppedCanvas.style.marginTop = '20px';
  // croppedCanvas.style.transform = `scale(1)`;
  // croppedCanvas.style.transformOrigin = `right top`;
  // document.getElementById('croppedSvg').appendChild(croppedCanvas);

  let luminanceSource = new HTMLCanvasElementLuminanceSource(croppedCanvas); // .invert() to support inverted codes

  const hybridBinarizer = new HybridBinarizer(luminanceSource);

  return new BinaryBitmap(hybridBinarizer);
};

There is also a very similar request: #356

Describe the solution you’d like I don’t really know how a good solution for this problem would look like in the component interface and also in the library beneath, since it also requires styling of the overlay and matching the displayed overlay with the cropped part.

Maybe it would be a good start to define custom hooks (callbacks) that can be implemented instead of overriding the complete functions.

Additional context

We also have to restart the scanner when the orientation changes in Safari, because the highlighted area does no longer match the croppedCanvas.

  @HostListener('window:orientationchange', ['$event'])
  onOrientationChange(event) {
     ...
  }

[shunta0009]

@NilsEngelbach thank you! I am now required to place a focusbox in the center of the ngx-scanner. I saw the code in “Define / Crop the area to search for codes” but I don’t know how to handle this code. Can you tell me more specifically?

[shreya-sawardekar]

@NilsEngelbach could you provide a code snippet on how to use the modifications you have mentioned? I am getting ‘BrowserCodeReader’ is deprecated.

I want to implement the cropping functionality you have mentioned.

Could you please help me out?