App: WSRTC integration

We have wsrtc as a WS-backed backend implementation. app can start to connect to it.

Most of that implementation is in world.js, which is currently backed by xrrtc, but those servers are not up and we should move to thew new API.

• connect/disconnect from the room
• send pose data
• send audio
• receive pose data
• receive audio

Local pose send

Local poses for the rig are grabbed and sent on the wire here: https://github.com/webaverse/app/blob/aeb45b6c3136bce4dfb1a64d06b420bf0f74a4b6/world.js#L133

This can already be done with pose is wsrtc, which has position/quaternion/scale tracking, but in app there is additional pose data for avatar states like animations, crouch, etc, and something called aux, which is all of the stuff attached to your avatar like wearables and pets. That can already be hooked up to use the wsrtc per-user metadata mechanism, but soon we’ll want to serialize this all to binary and transfer it as such in wsrtc, which depends on https://github.com/webaverse/wsrtc/issues/2.

Remote pose receive

Remote poses are received here: https://github.com/webaverse/app/blob/aeb45b6c3136bce4dfb1a64d06b420bf0f74a4b6/world.js#L212

This should be ported to get the pose from wsrtc.

Audio

We previously had a track-based low level audio solution, but now wsrtc gives us the audio context and the raw audio nodes which we can hook up in the graph to do positional audio for player.

Also note that wsrtc computes the volume levels, which are used for mouth flaps. That needs to also be plugged in: https://github.com/webaverse/app/blob/65db2efa3e89b695bb626f7f2cbcd4a078bf9dab/rig.js#L352

World

We can leave this to last, but wsrtc has a new Entity system which can replace the current “tracked objects”, and runtime can be moved to more of a component system responding to the Entity state.

This is still all based on y.js but I don’t think the app code will need to be aware of that anymore if it is using the wsrtc Entity abstraction.

[AGoblinKing]

https://developer.mozilla.org/en-US/docs/Web/API/Worker/Worker

A USVString representing the URL of the script the worker will execute. It must obey the same-origin policy.

ugh, that foils part of that plan.

[avaer]

There should be 1024b (1k) of extra data available for player poses now: https://github.com/webaverse/wsrtc/blob/198e1920c9f4b96d9a3ab1b11a64eaa466682847/wsrtc.js#L17

Any typed array is accepted as extra here, though we will need a serialization protocol for more complex data.