Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Batch Plugin - Struct access corruption in batch struct systems

See original GitHub issue

For some reason when dealing with struct based batched systems the batches seem to corrupt over time, its currently unknown why but @erinloy has made a reproduction here:

It takes a while to happen and has been observed in .net core 2 and 3, it seems to happen anywhere from 30 seconds to 3 minutes in. It is easier to see if you make it trigger on every update (60 times a second), and then after a certain period the structs start to read garbage data rather than the correct data, as shown here:

Ref 7 7348
Ref 8 7348
Str 12 1084
Str 13 140714692425167  <-- These numbers should be around 7000-8000 
Ref 9 7348
Str 14 1015
Str 15 1084
Str 16 140714692432655
Str 17 140714692432655
Str 18 1084

After a certain point it just blows up with a Fatal error. Internal CLR error. (0x80131506) which seems to indicate that memory is being corrupted at some point.

Behind the scenes structs are stored as arrays of components (i.e 1 array for each component type) and the batches effectively pins the memory for the arrays and then creates lookups for each component needed in the batch.

public Batch<T1, T2>[] Build(IReadOnlyList<IEntity> entities)
	var componentArray1 = ComponentDatabase.GetComponents<T1>(_componentTypeId1);
	var componentArray2 = ComponentDatabase.GetComponents<T2>(_componentTypeId2);
	var batches = new Batch<T1, T2>[entities.Count];

	fixed (T1* component1Handle = componentArray1)
	fixed (T2* component2Handle = componentArray2)
		for (var i = 0; i < entities.Count; i++)
			if (entities.Count != batches.Length)
			{ return new Batch<T1, T2>[0]; }
			var entity = entities[i];
			var component1Allocation = entity.ComponentAllocations[_componentTypeId1];
			var component2Allocation = entity.ComponentAllocations[_componentTypeId2];
			batches[i] = new Batch<T1, T2>(entity.Id, &component1Handle[component1Allocation], &component2Handle[component2Allocation]);

	return batches;

Once this has been done until the related entities/components change in any way it will keep re-using this batch so I am not sure if it is at some point moving the memory around after the lookup has been created so its now reading incorrect memory or if its something else.

I dont have much time to look into it at the moment but if anyone else has experience with unmanaged scenarios would be great to get some advice.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:11 (5 by maintainers)

github_iconTop GitHub Comments

Dorakucommented, Feb 6, 2020

I am probably a little crazy but where would be the fun in just using Unity’s ECS :p?

Dorakucommented, Jan 7, 2020

Your intuition is correct, you have no guaranty that the GC will not move around the array of components in memory, making your previously saved pointers in Batch items invalid. Using a pointer outside of its fixed scope creation leads to undefined behaviors, you are just lucky that it seemed to work for a while.

You could fix this issue by manually pinning the array without relying on the fixed scope so you keep it pinned longer.

component1Handle  = GCHandle.Alloc(componentArray1, GCHandleType.Pinned);

// to get your pointer
component1P = (T1*)component1Handle .AddrOfPinnedObject().ToPointer();

// to release the handle
if (component1Handle.IsAllocated)

Maybe wrap your Batch array in a custom type to also store the handles so you can release them correctly in a dispose pattern when you need to refresh the Batchs?

Note that your component arrays won’t be able to be moved around by the GC because of this, leaving you with a fragmented memory, it can be a double edge sword. Hopefully you already refresh your Batchs when the component array is resized for more entities too.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Android Studio error "Installed Build Tools revision 31.0.0 ...
Version 31.0.0 itself is corrupt, and we need to downgrade to 30. Steps I followed for the same: Open project structure → press...
Read more >
Batch Processing Massive Data Much Quicker with TiSpark
TiSpark is an Apache Spark plugin that works with the TiDB platform to answer complex online analytical processing (OLAP) queries.
Read more >
9 Fixes for Critical Structure Corruption BSOD Error in ...
9 Fixes for Critical Structure Corruption BSOD Error in Windows 10 · 9. Check System Files · 8. Hardware Testing · 7. Undo...
Read more >
Error conditions in Databricks
Unable to find batch <batchMetadataFile> . ... Please check that the specified table or struct exists and is accessible in the input columns ......
Read more >
Bulldozer: Batch Data Moving from Data Warehouse to ...
Bulldozer supports complicated schemas, like struct of struct type, array of struct, map of struct and map of map type. Data Version Control....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found