Three.js does not support the Windows Holographic platform

Description of the problem

Three.js does not support Windows Holographics. I’ve been working on a platform for creating JS apps in Microsoft HoloLens using WebGL. This requires a bunch of technology (old and new) to get working. I’m using Microsoft’s fork of ANGLE which has holographic support. I wrote my own WebGL implementation in C++ using ChakraCore as a bridge for the JS runtime. Currently, I have a working proof of concept app and I’d like to do something similar with Three.js. The issue I’m having is that Three.js of course doesn’t support the required GLSL property uHolographicViewProjectionMatrix which is provided by Microsoft’s ANGLE fork.

I suppose what I’m looking for is some guidance on how to achieve this with Three.js. We basically don’t have a need for a camera (HoloLens is a mobile platform so speed is critical, avoiding JS is necessary when dealing with the projection matrix to maximize hologram stability.)

Three.js version

https://github.com/lwansbrough/three.js/tree/holographic

Browser

None, see: https://github.com/lwansbrough/HolographicJS

OS

• Windows

Hardware Requirements (graphics card, VR Device, …)

• Microsoft HoloLens or HoloLens emulator

[Mugen87]

Closing. There is a basic three.js support for HoloJS, see https://github.com/Microsoft/HoloJS/pull/23

[carstenschwede]

I’m not sure how Microsoft ANGLE fork modifies the vertex shader, but at least for the “simple drawing mode” I understand that it should be sufficient to use a regular THREE.Camera with no special projection matrix and a modelView matrix based on the view matrix provided by the “Holographic App”. MS ANGLEs “magic” will double the drawing calls, rewind the view matrix transformation and apply view and projection matrices for each eye. I’m not sure how this is going to work out for more complex shader setups (e.g. it will expect the view matrix operation to be the last operation that is applied to vertices and assumes that the view matrix is only being used during the transformation of vertices but not in the fragment shader).

The “advanced drawing mode” expects you to provide instanced drawing calls for stereo rendering. The current THREE.StereoEffect works by switching cameras and re-rendering the whole scenery. If you’d like to go that route it might make sense to create some form THREE.InstancedStereoEffect first - although you would have to adapt THREE.WebGLRenderer and probably lots of the shaders as well.

The result is my render surface flashes different colours.

You are using simple mode right? Your current example uses a THREE.PerspectiveCamera with a non-identity projection matrix. You also use a THREE.MeshNormalMaterial which means the color in the fragment shader is affected based on the viewMatrix which is not “magically” corrected by ANGLE because it does not know about it. Does THREE.MeshBasicMaterial work as expected? I’ll see if I can get the HoloLens Emulator running and will give it a try.