TFJS WebGL on WebWorker still blocks GUI

After investigating a report by a user who was attempting to use web workers + tfjs to reduce lagging of GUI when model performs inference I have noticed that when using backend “webgl” web worker makes no effect on executing code in a separate context as this does not apply to GPU.

I have confirmed that by setting backend to “cpu” there is no performance issue, as the TFJS execution correctly is executed on a new thread, such that the browser DOM updates are not interfered with. I have also confirmed that browser relies on GPU to update DOM for anything visual - not CPU.

Thus the request here is how to limit “webgl” execution to leave enough processing for DOM updates for other user tasks to prevent this “jankiness” from occuring.

Demo of issue:

https://codepen.io/jasonmayes/project/editor/DBYaRj

Simply change first line of tfWorker.js to import one of:

importScripts('dist/cpu.js');

or for WebGL backend change to:

importScripts('dist/main.js');

Confirmed this issue across devices including Windows 10, Desktop, Chrome and also Xiaomi 8 (Android phone with Snapdragon 845 processor)

Example output from Chrome Dev tools shows GPU block when using WebGL backend and also that all execution comes from WebWorker.js.

[jasonmayes]

So the team have been researching into how to reduce GPU load when loading models / inference to give time back to browser render to reduce freezing of GUI etc. It seems to be more about how the browser schedules tasks to GPU. From what I understand WebWorker is aimed at CPU multithreading, and does not account for GPU tasks that are spawned from it which are still shared with the main browser level GPU render thread it seems. However focusing on GPU sharing instead we have had some promising results but still working on refining publishable solution to prod. This is actively under investigation though.

[jasonmayes]

I shall let @lina128 reply as she is currently working on the final solution to this which looks very promising in terms of that initial load blocking issue you mentioned by allowing time back to the browser render thread to do what it needs to do for updating GUI, and maybe she has some thoughts on q2 too.

Essentially though:

In the WebGL API, some methods are blocking, while others are not. These are different from the sync/async concept in JS. The blocking methods in WebGL mean that, certain WebGL entry points cause synchronous stalls on the calling thread (possibly the same calling thread Chrome uses for GUI rendering I believe as everything is rendered via GPU now on Chrome if I remember correctly). Even basic requests can take as long as 1ms, but they can take even longer if they need to wait for all graphics work to be completed.

In the new version Na is working on, we first compile and link everything without waiting because they are async, and then check and wait everything at the end, instead of individually, which gives time back to the browser to do other things it needs to perform in that time frame too.