Batch Plugin - Struct access corruption in batch struct systems

For some reason when dealing with struct based batched systems the batches seem to corrupt over time, its currently unknown why but @erinloy has made a reproduction here:

https://github.com/erinloy/EcsRxStructTest

It takes a while to happen and has been observed in .net core 2 and 3, it seems to happen anywhere from 30 seconds to 3 minutes in. It is easier to see if you make it trigger on every update (60 times a second), and then after a certain period the structs start to read garbage data rather than the correct data, as shown here:

Ref 7 7348
Ref 8 7348
Str 12 1084
Str 13 140714692425167  <-- These numbers should be around 7000-8000 
Ref 9 7348
Str 14 1015
Str 15 1084
Str 16 140714692432655
Str 17 140714692432655
Str 18 1084

After a certain point it just blows up with a Fatal error. Internal CLR error. (0x80131506) which seems to indicate that memory is being corrupted at some point.

Behind the scenes structs are stored as arrays of components (i.e 1 array for each component type) and the batches effectively pins the memory for the arrays and then creates lookups for each component needed in the batch.

public Batch<T1, T2>[] Build(IReadOnlyList<IEntity> entities)
{
	var componentArray1 = ComponentDatabase.GetComponents<T1>(_componentTypeId1);
	var componentArray2 = ComponentDatabase.GetComponents<T2>(_componentTypeId2);
	
	var batches = new Batch<T1, T2>[entities.Count];

	fixed (T1* component1Handle = componentArray1)
	fixed (T2* component2Handle = componentArray2)
	{
		for (var i = 0; i < entities.Count; i++)
		{
			if (entities.Count != batches.Length)
			{ return new Batch<T1, T2>[0]; }
			
			var entity = entities[i];
			var component1Allocation = entity.ComponentAllocations[_componentTypeId1];
			var component2Allocation = entity.ComponentAllocations[_componentTypeId2];
			batches[i] = new Batch<T1, T2>(entity.Id, &component1Handle[component1Allocation], &component2Handle[component2Allocation]);
		}
	}

	return batches;
}

Once this has been done until the related entities/components change in any way it will keep re-using this batch so I am not sure if it is at some point moving the memory around after the lookup has been created so its now reading incorrect memory or if its something else.

I dont have much time to look into it at the moment but if anyone else has experience with unmanaged scenarios would be great to get some advice.

[Doraku]

I am probably a little crazy but where would be the fun in just using Unity’s ECS :p?

[Doraku]

Your intuition is correct, you have no guaranty that the GC will not move around the array of components in memory, making your previously saved pointers in Batch items invalid. Using a pointer outside of its fixed scope creation leads to undefined behaviors, you are just lucky that it seemed to work for a while.

You could fix this issue by manually pinning the array without relying on the fixed scope so you keep it pinned longer.

component1Handle  = GCHandle.Alloc(componentArray1, GCHandleType.Pinned);

// to get your pointer
component1P = (T1*)component1Handle .AddrOfPinnedObject().ToPointer();

// to release the handle
if (component1Handle.IsAllocated)
{
    component1Handle.Free();
}

Maybe wrap your Batch array in a custom type to also store the handles so you can release them correctly in a dispose pattern when you need to refresh the Batchs?

Note that your component arrays won’t be able to be moved around by the GC because of this, leaving you with a fragmented memory, it can be a double edge sword. Hopefully you already refresh your Batchs when the component array is resized for more entities too.