Substantial performance issues in latest JS, possibly spawning at 0,0,0 in multiplayer

What you were trying to do

Play test recent JoshariasSurvival with @ktksan’s awesome new GSOC biome stuff, including enabling of WildAnimals and some temperature maps.

After initial play testing also lagged wildly at times even in single player https://github.com/Terasology/WildAnimals/pull/58 was submitted to cut excessive deer spawn. This appears to have helped some

Timeouts may also have been helped some by #4113 - you really need to be patient in some cases trying to join!

What actually happened

We had a variety of problems just trying to get a server stable, extensive load times, timeout failures trying to spawn in the world, massive lag spikes on the server with players trying to join/leave, to the point where a hosting client might completely stall for a minute or two while a client tries to join.

A weird consequence appears to be when you do finally spawn you might be embedded deep in the ground on the client’s view of the world - but the server thinks you’re perfectly fine and happy at spawn, able to move around and everything (as observed by a host player) despite not moving at all on the client (because stuck in solid ground). Maybe this could be a desync between host and client, with the client having partially loaded without blocks to land on, falling for a while until chunks caught up and froze the player in place?

Edit: Got more info on that case, the 2nd player appeared stuck at exactly 0, 0, 0 - we’ve had bugs like that in the past. Something about a LocationComponent not initializing or attaching right to the 2nd player?

How to reproduce

• Unsure - but seen both from source playing JS lately, as well as while testing Omega dev build 1094.
• Host a multiplayer game of JS and survive loading it (may take a while)
• Try to join with a second client - may take longer, may freeze the host for a period of time
• After a long freeze for the host player everything may then work perfectly fine, no lag, can see the 2nd client fine, despite that player being embedded deep in the ground in their game window

Log details and game version

Omega dev build 1094

Some log snippets from near the end of the 2nd client’s log (host log was unremarkable) - highlights:

• Weird “unavailable entity” logs
• Odd warnings from DAG node stuff
• ClimateMapDisplaySystem is part of ktksan’s changes and may just not be wired up right for multiplayer yet

...
21:48:34.731 [main] INFO  o.t.e.prefab.internal.PrefabFormat - Attempting to deserialize prefab JoshariasSurvival:AssembleCrudeHammerFromRock with inputs [/JoshariasSurvival/assets/prefabs/AssembleCrudeHammerFromRock.prefab]
21:48:34.732 [main] INFO  o.t.e.prefab.internal.PrefabFormat - Attempting to deserialize prefab PlantPack:Oreo with inputs [/PlantPack/assets/prefabs/crop/Oreo.prefab]
21:48:34.732 [main] ERROR o.t.p.serializers.PrefabSerializer - Prefab contains unknown component 'ChangingBlocks'
21:48:34.732 [main] INFO  o.t.e.prefab.internal.PrefabFormat - Attempting to deserialize prefab AdditionalFruits:WorkstationRecipe with inputs [/AdditionalFruits/assets/prefabs/WorkstationRecipe.prefab]
21:48:34.732 [main] ERROR o.t.p.serializers.PrefabSerializer - Prefab contains unknown component 'ListRecipes'
21:48:34.748 [main] INFO  o.t.w.c.b.ExtraBlockDataManager - Extra data slots registered: BiomesAPI.biomeHash -> 0
21:48:35.890 [main] ERROR o.t.engine.ComponentSystemManager - Cannot load ModuleRendering, must be a subclass of ComponentSystem
21:48:36.583 [main] WARN  o.t.engine.internal.TimeBase - Delta too great (1834), capping to 1000
21:48:36.745 [main] INFO  o.t.r.world.WorldRendererImpl - 
Initialising rendering class CoreRenderingModule from CoreRendering module.

21:48:36.762 [main] INFO  o.t.rendering.dag.RenderGraph - Warning, Output:CoreRendering:backdropNode:BufferPair1(connected to CoreRendering:opaqueObjectsNode:BufferPair1, ;)connection is already read somewhere else.
21:48:36.765 [main] INFO  o.t.rendering.dag.RenderGraph - Warning, Output:CoreRendering:backdropNode:BufferPair1(connected to CoreRendering:opaqueBlocksNode:BufferPair1, CoreRendering:opaqueObjectsNode:BufferPair1, ;)connection is already read somewhere else.
21:48:36.767 [main] INFO  o.t.rendering.dag.RenderGraph - Warning, Output:CoreRendering:backdropNode:BufferPair1(connected to CoreRendering:alphaRejectBlocksNode:BufferPair1, CoreRendering:opaqueBlocksNode:BufferPair1, CoreRendering:opaqueObjectsNode:BufferPair1, ;)connection is already read somewhere else.
21:48:36.793 [main] INFO  o.t.rendering.dag.RenderGraph - Warning, Output:CoreRendering:applyDeferredLightingNode:BufferPair1(connected to CoreRendering:outlineNode:BufferPair1, ;)connection is already read somewhere else.
21:48:36.796 [main] INFO  o.t.rendering.dag.RenderGraph - Warning, Output:CoreRendering:applyDeferredLightingNode:BufferPair1(connected to CoreRendering:chunksRefractiveReflectiveNode:BufferPair1, CoreRendering:outlineNode:BufferPair1, ;)connection is already read somewhere else.
21:48:36.827 [main] WARN  o.t.rendering.dag.RenderGraph - Warning, Output:CoreRendering:toneMappingNode:FBO1(connected to CoreRendering:firstLateBlurNode:FBO1, ;)connection is already read somewhere else.
21:48:36.833 [main] INFO  o.t.r.world.WorldRendererImpl - 
Initialising rendering class AdvancedRenderingModule from CoreRendering module.

21:48:36.837 [main] INFO  o.t.rendering.dag.RenderGraph - Warning, Output:CoreRendering:backdropNode:BufferPair1(connected to CoreRendering:alphaRejectBlocksNode:BufferPair1, CoreRendering:overlaysNode:BufferPair1, CoreRendering:opaqueBlocksNode:BufferPair1, CoreRendering:opaqueObjectsNode:BufferPair1, ;)connection is already read somewhere else.
21:48:36.845 [main] WARN  o.t.rendering.dag.RenderGraph - Trying to disconnect two nodes that aren't connected, CoreRendering:backdropNode and CoreRendering:opaqueObjectsNode
21:48:36.845 [main] WARN  o.t.rendering.dag.RenderGraph - WARNING: destination connection (Output:CoreRendering:finalHazeNode:BufferPair1(connected to CoreRendering:opaqueObjectsNode:BufferPair1, ;)) is already connected to ({CoreRendering:opaqueObjectsNode:BufferPair1=Input:CoreRendering:opaqueObjectsNode:BufferPair1(connected to CoreRendering:finalHazeNode:BufferPair1, ;)}
21:48:36.845 [main] WARN  o.t.rendering.dag.RenderGraph - Trying to disconnect two nodes that aren't connected, CoreRendering:backdropNode and CoreRendering:opaqueBlocksNode
21:48:36.845 [main] WARN  o.t.rendering.dag.RenderGraph - WARNING: destination connection (Output:CoreRendering:finalHazeNode:BufferPair1(connected to CoreRendering:opaqueBlocksNode:BufferPair1, CoreRendering:opaqueObjectsNode:BufferPair1, ;)) is already connected to ({CoreRendering:opaqueBlocksNode:BufferPair1=Input:CoreRendering:opaqueBlocksNode:BufferPair1(connected to CoreRendering:finalHazeNode:BufferPair1, ;), CoreRendering:opaqueObjectsNode:BufferPair1=Input:CoreRendering:opaqueObjectsNode:BufferPair1(connected to CoreRendering:finalHazeNode:BufferPair1, ;)}
21:48:36.845 [main] WARN  o.t.rendering.dag.RenderGraph - Trying to disconnect two nodes that aren't connected, CoreRendering:backdropNode and CoreRendering:alphaRejectBlocksNode
21:48:36.845 [main] WARN  o.t.rendering.dag.RenderGraph - WARNING: destination connection (Output:CoreRendering:finalHazeNode:BufferPair1(connected to CoreRendering:alphaRejectBlocksNode:BufferPair1, CoreRendering:opaqueBlocksNode:BufferPair1, CoreRendering:opaqueObjectsNode:BufferPair1, ;)) is already connected to ({CoreRendering:alphaRejectBlocksNode:BufferPair1=Input:CoreRendering:alphaRejectBlocksNode:BufferPair1(connected to CoreRendering:finalHazeNode:BufferPair1, ;), CoreRendering:opaqueBlocksNode:BufferPair1=Input:CoreRendering:opaqueBlocksNode:BufferPair1(connected to CoreRendering:finalHazeNode:BufferPair1, ;), CoreRendering:opaqueObjectsNode:BufferPair1=Input:CoreRendering:opaqueObjectsNode:BufferPair1(connected to CoreRendering:finalHazeNode:BufferPair1, ;)}
21:48:36.845 [main] WARN  o.t.rendering.dag.RenderGraph - Trying to disconnect two nodes that aren't connected, CoreRendering:backdropNode and CoreRendering:overlaysNode
21:48:36.855 [main] INFO  o.t.rendering.dag.RenderGraph - Warning, Output:CoreRendering:applyDeferredLightingNode:BufferPair1(connected to CoreRendering:chunksRefractiveReflectiveNode:BufferPair1, CoreRendering:prePostCompositeNode:BufferPair1, CoreRendering:outlineNode:BufferPair1, ;)connection is already read somewhere else.
21:48:36.861 [main] INFO  o.t.rendering.dag.RenderGraph - Warning, Output:CoreRendering:simpleBlendMaterialsNode:BufferPair1(connected to CoreRendering:initialPostProcessingNode:BufferPair1, ;)connection is already read somewhere else.
21:48:36.864 [main] INFO  o.t.rendering.dag.RenderGraph - Warning, Output:CoreRendering:simpleBlendMaterialsNode:BufferPair1(connected to CoreRendering:initialPostProcessingNode:BufferPair1, CoreRendering:lightShaftsNode:BufferPair1, ;)connection is already read somewhere else.
21:48:37.057 [main] INFO  o.t.p.updating.ParticleUpdaterImpl - Registering GeneratorFunction for Component class class org.terasology.particles.components.generators.VelocityRangeGeneratorComponent
21:48:37.057 [main] INFO  o.t.p.updating.ParticleUpdaterImpl - Registering GeneratorFunction for Component class class org.terasology.particles.components.generators.EnergyRangeGeneratorComponent
21:48:37.058 [main] INFO  o.t.p.updating.ParticleUpdaterImpl - Registering GeneratorFunction for Component class class org.terasology.particles.components.generators.ColorRangeGeneratorComponent
21:48:37.058 [main] INFO  o.t.p.updating.ParticleUpdaterImpl - Registering GeneratorFunction for Component class class org.terasology.particles.components.generators.PositionRangeGeneratorComponent
21:48:37.058 [main] INFO  o.t.p.updating.ParticleUpdaterImpl - Registering GeneratorFunction for Component class class org.terasology.particles.components.generators.TextureOffsetGeneratorComponent
21:48:37.058 [main] INFO  o.t.p.updating.ParticleUpdaterImpl - Registering GeneratorFunction for Component class class org.terasology.particles.components.generators.ScaleRangeGeneratorComponent
21:48:37.058 [main] INFO  o.t.p.updating.ParticleUpdaterImpl - Registering AffectorFunction for Component class class org.terasology.particles.components.affectors.AccelerationAffectorComponent
21:48:37.058 [main] INFO  o.t.p.updating.ParticleUpdaterImpl - Registering AffectorFunction for Component class class org.terasology.particles.components.affectors.VelocityAffectorComponent
21:48:37.059 [main] INFO  o.t.i18n.TranslationSystemImpl - Discovered engine:menu_hi
...
21:48:37.068 [main] INFO  o.t.i18n.TranslationSystemImpl - Discovered engine:menu_ru
21:48:37.254 [main] INFO  o.terasology.minion.work.WorkFactory - Initialize WorkFactory
21:48:37.261 [main] INFO  org.terasology.minion.work.WorkBoard - Initialize WorkBoard
21:48:37.264 [main] ERROR o.t.engine.ComponentSystemManager - Failed to initialise system org.terasology.climateConditions.visualization.ClimateMapDisplaySystem@2c1eba2e
java.lang.NullPointerException: null
	at org.terasology.climateConditions.visualization.ClimateMapDisplaySystem.setClimateSeed(ClimateMapDisplaySystem.java:45)
	at org.terasology.climateConditions.visualization.ClimateMapDisplaySystem.initialise(ClimateMapDisplaySystem.java:34)
	at org.terasology.engine.ComponentSystemManager.initialiseSystem(ComponentSystemManager.java:164)
	at org.terasology.engine.ComponentSystemManager.initialise(ComponentSystemManager.java:148)
	at org.terasology.engine.modes.loadProcesses.InitialiseSystems.step(InitialiseSystems.java:51)
	at org.terasology.engine.modes.StateLoading.update(StateLoading.java:236)
	at org.terasology.engine.TerasologyEngine.tick(TerasologyEngine.java:468)
	at org.terasology.engine.TerasologyEngine.mainLoop(TerasologyEngine.java:431)
	at org.terasology.engine.TerasologyEngine.run(TerasologyEngine.java:407)
	at org.terasology.engine.Terasology.main(Terasology.java:182)
21:48:37.345 [main] INFO  o.terasology.biomesAPI.BiomeManager - Registered biome CoreWorlds:MOUNTAINS with id 3084
21:48:37.345 [main] INFO  o.terasology.biomesAPI.BiomeManager - Registered biome CoreWorlds:SNOW with id -7875
21:48:37.345 [main] INFO  o.terasology.biomesAPI.BiomeManager - Registered biome CoreWorlds:DESERT with id -25265
21:48:37.345 [main] INFO  o.terasology.biomesAPI.BiomeManager - Registered biome CoreWorlds:FOREST with id -15017
21:48:37.345 [main] INFO  o.terasology.biomesAPI.BiomeManager - Registered biome CoreWorlds:OCEAN with id -16252
21:48:37.346 [main] INFO  o.terasology.biomesAPI.BiomeManager - Registered biome CoreWorlds:BEACH with id 26857
21:48:37.346 [main] INFO  o.terasology.biomesAPI.BiomeManager - Registered biome CoreWorlds:PLAINS with id 18723
21:48:56.397 [main] INFO  o.t.l.p.ThirdPersonRemoteClientSystem - Somehow ended up in the else during linkHeldItemLocationForRemotePlayer - current item was EntityRef{id=0} and new item EntityRef{id=0}
21:49:29.779 [main] WARN  o.t.engine.internal.TimeBase - Delta too great (33421), capping to 1000
21:49:30.383 [main] INFO  o.t.l.p.ThirdPersonRemoteClientSystem - Somehow ended up in the else during linkHeldItemLocationForRemotePlayer - current item was EntityRef{id=0} and new item EntityRef{id=0}
21:49:30.383 [main] INFO  o.t.network.internal.ServerImpl - Dropping event PlaySoundEvent for unavailable entity EntityRef{id=0}
21:49:30.383 [main] INFO  o.t.network.internal.ServerImpl - Dropping event PlaySoundEvent for unavailable entity EntityRef{id=0}
21:49:30.396 [main] INFO  o.t.network.internal.ServerImpl - Destroying entity: EntityRef{id = 1159, netId = 818, prefab = 'CoreAssets:defaultBlockParticles'}
21:49:30.503 [main] INFO  o.t.network.internal.ServerImpl - Dropping event PlaySoundEvent for unavailable entity EntityRef{id=0}
21:49:30.504 [main] INFO  o.t.network.internal.ServerImpl - Dropping event PlaySoundEvent for unavailable entity EntityRef{id=0}
21:49:30.505 [main] INFO  o.t.network.internal.ServerImpl - Dropping event PlaySoundEvent for unavailable entity EntityRef{id=0}
21:49:30.505 [main] INFO  o.t.network.internal.ServerImpl - Dropping event PlaySoundEvent for unavailable entity EntityRef{id=0}
21:49:30.506 [main] INFO  o.t.network.internal.ServerImpl - Destroying entity: EntityRef{id = 1162, netId = 821, prefab = 'CoreAssets:defaultBlockParticles'}
21:49:30.506 [main] INFO  o.t.network.internal.ServerImpl - Destroying entity: EntityRef{id = 1165, netId = 825, prefab = 'ManualLabor:PlantMatter'}
21:49:30.507 [main] INFO  o.t.network.internal.ServerImpl - Destroying entity: EntityRef{id = 1164, netId = 823, prefab = 'CoreAssets:defaultBlockParticles'}
21:49:30.508 [main] INFO  o.t.network.internal.ServerImpl - Dropping event PlaySoundEvent for unavailable entity EntityRef{id=0}
21:49:30.508 [main] INFO  o.t.network.internal.ServerImpl - Dropping event PlaySoundEvent for unavailable entity EntityRef{id=0}
21:49:30.508 [main] INFO  o.t.network.internal.ServerImpl - Destroying entity: EntityRef{id = 1168, netId = 828, prefab = 'ManualLabor:PlantMatter'}
21:49:30.510 [main] INFO  o.t.network.internal.ServerImpl - Destroying entity: EntityRef{id = 1167, netId = 826, prefab = 'CoreAssets:defaultBlockParticles'}
21:49:34.292 [main] INFO  o.t.network.internal.ServerImpl - Destroying entity: EntityRef{id = 1175, netId = 829, prefab = 'CoreAssets:defaultBlockParticles'}

Screenie

Top left is the hosting player, and everything is fine. Looking at player 2 who on that screen moves around perfectly normally despite being hopelessly stuck in the ground on their own screen on the bottom right.

[keturn]

in fact, a game with all of those ManualLabor dependencies except for ManualLabor itself seems to not show the issue.

So I guess it’s something about ManualLabor?

I still don’t know what that would be. I reviewed the recent commits there and there are some new textures and a few very small changes for TeraNUI.

[keturn]

Some findings:

A big source of slowness in sendInitialEntities is when it was serializing InventoryAccessComponent. It uses this getAsString method:

https://github.com/MovingBlocks/Terasology/blob/012985ccb97b6872af9d5bfdb3c589845ab98605/engine/src/main/java/org/terasology/persistence/typeHandling/extensionTypes/ComponentClassTypeHandler.java#L31-L32

resulting in ModuleEnvironmentSandbox.getSubTypeIdentifier running loadClass on all Component classes.

To some extent, this gets worse with the number of Components loaded in the type registry, so it’s slower with bigger module sets like JS.

I am not sure exactly what it is about Manual Labor in particular that sets this off. Its HumanMachineAuthoritySystem does add a Component to the Player entity, so maybe that leads to more Components being serialized when a new Player connects? but I can’t connect the dots from there to InventoryAccessComponent.

and I still don’t know why this is something that would have become so much worse over the past month or two.