question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Implement native tf.browser.fromPixels() for WASM backend

See original GitHub issue

Currently tf.browser.fromPixels() has native implementation for WebGL backend while WASM falls back to default canvas.getContext('2d').getImageData(0, 0, canvas.width, canvas.height).data;

This means it will: a) Fail if input already has a different context
e.g., canvas with webgl context because user performed some image pre-processing b) It is really slow

On FullHD input, WebGL native implementation executes in near 0ms while getImageData() takes ~20ms

This significantly impacts overall performance of WASM backend in general.

Example, if model execution takes 33ms in both WebGL and WASM, frame rate in WebGL will be 30 FPS, but in WASM it will be 1000/(20+33) = 18 FPS - that is 36% performance difference just because of lack of native implementation to get image data.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:2
  • Comments:10 (8 by maintainers)

github_iconTop GitHub Comments

1reaction
axingingcommented, Jun 8, 2022

@vladmandic , so do you agree that we turn flag CANVAS2D_WILL_READ_FREQUENTLY to true by default?

BTW, I did more investigation on wasm fromPixels. There are two fromPixels bottlenecks on wasm: getImageData and data copy (from 4 channels to three channels). A typical measured times for these bottlenecks(720p image) are: getImageData: 2.52 ms, data copy: 8.80 ms

It seems when image gets larger, more efforts are spent on data copy. And I think data copy may possibly be improved by wasm.

I wrote a simple case to prove this: https://github.com/axinging/asyncawaitdemo/blob/main/webgl/index_canvas_bug2.html

1reaction
axingingcommented, Jun 2, 2022

Hi, @rdutta1999 @vladmandic Could you please try this PR(https://github.com/tensorflow/tfjs/pull/6445): During my test, set CANVAS2D_WILL_READ_FREQUENTLY to true will greatly improve getImageData perf. You can do below before run your wasm case:

tf.env().set(‘CANVAS2D_WILL_READ_FREQUENTLY’, true)

Read more comments on GitHub >

github_iconTop Results From Across the Web

Introducing the WebAssembly backend for TensorFlow.js
We're happy to announce that TensorFlow.js now provides a WebAssembly (WASM) backend for both the browser and for Node.js!
Read more >
Tensorflow.js available on WebAssembly backend
Next step is to let Parcel know how to load the WASM file. When the WASM backend is initialized, there will be a...
Read more >
tensorflow/tfjs-backend-wasm: Docs & Community
This package adds a WebAssembly backend to TensorFlow.js. It currently supports the following models from our models repo:.
Read more >
tensorflow/tfjs-core
@tensorflow/tfjs-core/dist/tf-core.js.map ... n */\nexport class KernelBackend implements TensorStorage, Backend, BackendTimer ... floatPrecision() === 32 ?
Read more >
TensorFlow.js: Make your own "Teachable Machine" using ...
... transfer learning is very well suited for the web browser ... that any user can use to recognize a custom object with...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found