Implement native tf.browser.fromPixels() for WASM backend
See original GitHub issueCurrently tf.browser.fromPixels()
has native implementation for WebGL backend while WASM falls back to default canvas.getContext('2d').getImageData(0, 0, canvas.width, canvas.height).data;
This means it will:
a) Fail if input already has a different context
e.g., canvas with webgl
context because user performed some image pre-processing
b) It is really slow
On FullHD input, WebGL native implementation executes in near 0ms while getImageData()
takes ~20ms
This significantly impacts overall performance of WASM backend in general.
Example, if model execution takes 33ms in both WebGL and WASM, frame rate in WebGL will be 30 FPS, but in WASM it will be 1000/(20+33) = 18 FPS - that is 36% performance difference just because of lack of native implementation to get image data.
Issue Analytics
- State:
- Created 3 years ago
- Reactions:2
- Comments:10 (8 by maintainers)
Top GitHub Comments
@vladmandic , so do you agree that we turn flag CANVAS2D_WILL_READ_FREQUENTLY to true by default?
BTW, I did more investigation on wasm fromPixels. There are two fromPixels bottlenecks on wasm: getImageData and data copy (from 4 channels to three channels). A typical measured times for these bottlenecks(720p image) are: getImageData: 2.52 ms, data copy: 8.80 ms
It seems when image gets larger, more efforts are spent on data copy. And I think data copy may possibly be improved by wasm.
I wrote a simple case to prove this: https://github.com/axinging/asyncawaitdemo/blob/main/webgl/index_canvas_bug2.html
Hi, @rdutta1999 @vladmandic Could you please try this PR(https://github.com/tensorflow/tfjs/pull/6445): During my test, set CANVAS2D_WILL_READ_FREQUENTLY to true will greatly improve getImageData perf. You can do below before run your wasm case:
tf.env().set(‘CANVAS2D_WILL_READ_FREQUENTLY’, true)