Overhaul generation of bindings layer

The “bindings” layer that allows our C++ plugin to call into Unity and the .NET platform is currently generated by a fork of UnityNativeScripting. UnityNativeScripting is a fantastic idea, it’s backed up by really useful tech articles, and is a solid proof-of-concept implementation. To take Cesium for Unity to production, however, we need something a bit more production-ready. Some of the problems with it:

1. Lots of bugs and shortcomings. I’ve had to make significant changes to it to support the Cesium for Unity prototype so far.
2. The code generator is integrated into Unity, which makes it awkward to run outside of Unity as a batch job.
3. The code generator is a single source file with over 13k lines of code, which makes it hard to understand and modify. A lot of this comes from generating code by repeatedly calling Append on a StringBuilder, rather than using templating.
4. Its hot-reloading system is unlikely to ever work for us, because it requires all C++ allocations to happen in a designated C#-owned buffer. And it also seems to be getting in our way. Anytime Unity tries to hot-reload anything (even if it’s game code not plugin code being reloaded), we get into an inconsistent state that causes errors and crashes. We can live without hot-reloading of our plugin if necessary, but we can’t tell our users that they’re not allowed to hot-reload their own code.
5. It uses a custom system to allow the C++ side to “reference” C# objects. This system requires designating a maximum number of objects up-front. And it’s mostly unncessary, because .NET’s built-in GCHandle does this already.
6. UnityNativeSharp allows a C++ class to derive from a C# class. So, for example, we can implement a MonoBehaviour in C++. However, it doesn’t handle the ownership/lifetime in a principled way, leading to hard-to-debug gotchas. For example, you can’t create an instance of MyCPPMonoBehaviour and attach it to a GameObject without carefully ensuring that the C++ class instance sticks around until the GameObject is destroyed or the MonoBehaviour is removed. Otherwise you’ll end up with a broken MonoBehaviour attached to the GameObject. This means the rules for using C++ objects derived from C# classes are different from the rules for using regular C# classes, which is super confusing. I believe the right way to solve this is to separate the “C++ wrapper class” from the “C++ implementation class.” The C++ wrapper instance simply refers to the C# instance and allows it to be used from C++, just like any wrapper for any C# object. The C++ implementation instance is created by and owned by the C# instance, and allows for the implementation of parts of the C# class to be delegated to C++.
7. The types, properties, and methods to expose to C++ are defined in a JSON file, but writing that JSON file is more difficult than it needs to be. The syntax is confusing and inconsistent, particularly for generic types. And if you want to expose a particular class, you also need to manually expose every base class or interface implemented by that class. Similarly, if a method returns a particular type, you have to manually expose that type. The generator could fairly easily automatically discover the set of all types that need to be exposed from a much smaller user-specified set.

[kring]

Whoo hoo the prototype is working, running on the new bindings layer!

[kring]

Events and Delegates are tricky. UnityNativeScripting’s approach works, but it gets the semantics wrong (IMO). Its approach is described here: https://www.jacksondunstan.com/articles/4174 The problem is in how it manages lifetime. In normal C#, it works like this. If I have an object A which has an event or delegate field MyDelegate, and I assign it a delegate pointing to a method DoStuff on my object B, like this:

A.MyDelegate = B.DoStuff;

Then B won’t get garbage collected as long as A is ineligible for garbage collection and that delegate instance continues to exist. That’s true even if no one else in the entire app has a reference to B.

On the other hand, the existence of this delegate has no impact on the lifetime of A. If no references to A exists anywhere, then A will be garbage collected, as will myDelegate. If no other references to B exist (other than the delegate), then B will be garbage collected too.

This is all sort of obvious if you understand how delegates work. A delegate instance is really nothing more than a class instance with a target object field and a target method field. However, it often trips up people who are new to C#.

Now extending this to C++, we should be able to create a delegate around a C++ std::function or class instance, and that function or class instance should be kept alive for as long as the delegate is kept alive. With UnityNativeScripting, however, the C++ object’s lifetime must be controlled explicitly, and when it is destroyed the delegate effectively becomes inert. Invoking it does nothing.

This is confusing, and attempts to work around it can easily lead to memory leaks.

Consistent with out how we handle C# classes with some of their methods implemented in C++, we should aim to make the semantics match those in C#.

To achieve this, we need:

1. A delegate (e.g. Action) constructor that takes a std::function.
2. The constructor must copy/move the std::function instance into a heap-allocated function. We can do fancier things like pooling, but this is the simplest thing.
3. Call a delegate create function on the C# side, passing it a void* to the std::function.
4. The C# code creates an instance of a class generated for this purpose, e.g. ActionNative. The instance holds onto that void* from the C++ side, and has a Dispose method and a finalizer to delete it at the appropriate time.
5. ActionNative has an Invoke method, and a regular Action delegate is created pointing to that Invoke method. A handle to this delegate is returned to the C++ code.
6. When the delegate is invoked, the Invoke method calls back into the C++ code, passing the void* to the std::function. The C++ code casts the void* back to a std::function* and invokes the function.

One thing that’s not great about this plan is that the std::functions will only ever be freed by the finalizer, because C# doesn’t have any pattern for explicitly disposing delegates. As long as we’re not creating them rapid-fire this should be fine, though. If it does turn out to be a problem, we can probably optimize for certain cases, e.g. provide an explicit dispose on the C++ side, or dispose when the delegate is reassigned null.