Kafka in Unity not working. Seems to have broken assemblies or conflicting with Unity's. How to fix?

Description

I want to integrate Kafka in Unity for a Windows build (I am on Windows).

However I either get errors such as this one:

Error: Could not load signature of Confluent.Kafka.IAsyncDeserializer`1[T]:DeserializeAsync due to: Could not load file or assembly 'System.Memory, Version=4.0.1.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. assembly:System.Memory, Version=4.0.1.0,

Or if I install the System.Memory version that comes with the Kafka NuGet dependency, I get errors all over the place of Unity’s own method/class references no longer being found.

How to reproduce

I have tried installing the latest release (1.3.0) from nuget (using the UnityPackage ‘NuGetForUnity’). This did not seem to work and had the second type of error result described above.

I have then tried to build the source and copy the relevant DLL’s manually, and tried the same with the nuget release version. This gives the first type of error result described above.

Can anyone please help me with installing Kafka for Unity?

[Mason-McGough]

@ztolley I needed to do the same thing and recently figured it out. You may be missing the librdkafka.dll file for your system. For reference, here are the steps I used to get Kafka working in my project:

1. Download and install the NuGet for Unity .unitypackage from their GitHub.
2. In Unity, open “NuGet”->“Manage NuGet Packages” and install “Confluent.Kafka”.
3. From the NuGet package page for librdkafka, click “Download Package”.
4. Extract the archive. Under the “runtimes” folder, copy the folder for your system (I am on Ubuntu 18.04 so I picked “linux-x64”) and paste it under “Assets/Packages/Confluent.Kafka.<version>/lib/netstandard2.0” in Unity.
5. Create a C# script called “KafkaConsumer” and paste the code @Phantomb provided.
6. If needed, change BootstrapServers = "localhost:9092" to reflect your server and c.Subscribe("networktopic_data"); to the name of the topic you wish to subscribe to.
7. You might need to modify a few more things to get it to work for your case. To get started, find and replace StreamMessage with string for now. Replace StreamMessage message = ParseStreamMessage.Decode(cr.Value); with string message = Encoding.UTF8.GetString(cr.Value);. Replace ProcessPBMessages.Process(message); with Debug.Log(message);. Once you can play the scene and confirm you are receiving messages in the Console, you can try using Newtonsoft JsonConvert or something similar to work for your use case.

Hope that helps!

[mhowlett]

reopening, it would be good for us to test this and make this work ootb.