Errors while trying to run this using .NET 6 arm64 SDK with M1 Macbook Pro

Description

Im getting an error when trying to use this inside .net 6 web project on M1 macbook pro (arm64 architecture). The error is: System.DllNotFoundException: Failed to load the librdkafka native library. at Confluent.Kafka.Impl.Librdkafka.TrySetDelegates(List1 nativeMethodCandidateTypes) at Confluent.Kafka.Impl.Librdkafka.LoadOSXDelegates(String userSpecifiedPath) at Confluent.Kafka.Impl.Librdkafka.Initialize(String userSpecifiedPath) at Confluent.Kafka.Consumer2…ctor(ConsumerBuilder2 builder) at Confluent.Kafka.ConsumerBuilder2.Build()

It works if i run the project on older .net5 using rosetta, without any code change. If i update to .net6 and use arm, it gives this error on first consumer/producer usage. I have tried side loading the library from “brew install librdkafka” using Library.Load, but no success there. Any chance to get it to work on .net 6, or is it not supported yet?

How to reproduce

I think description says it all, just try to install only .net6 sdk on m1 macbook and try to run this nuget

Checklist

Please provide the following information:

• [x ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [ 1.8.2] Confluent.Kafka nuget version.
• [1.8.2] Apache Kafka version.
• [ x] Client configuration.
• [ MacOs Monterey] Operating system.
• [description ] Provide logs (with “debug” : “…” as necessary in configuration).
• [ x] Provide broker log excerpts.
• [ for me, yes, for others, probably not so critical] Critical issue.

[k-kapsazov]

Hi @vijaybw , The files you need are:

librdkafka++.1.dylib
librdkafka++.dylib
librdkafka.a
librdkafka++.a
librdkafka.1.dylib
librdkafka.dylib

After you add them to your project the file structure should look like this:

Don’t forget to mark all added files as “Content” and “Copy always” in your IDE. In Rider or Visual Studio that is done by right click on given file > Properties > options “Build action” and “Copy to output directory”.

That is all you need. Now when you build the project librdkafka should work.

[k-kapsazov]

Hi, The other day my team hit the same problem. After some digging in the code we found the actual error is:

Unable to load shared library 'librdkafka' or one of its dependencies. In order to help diagnose loading problems, 
consider setting the DYLD_PRINT_LIBRARIES environment variable:
 dlopen(liblibrdkafka, 0x0001): tried: 'liblibrdkafka' (no such file), 
 '/usr/local/lib/liblibrdkafka' (no such file), 
 '/usr/lib/liblibrdkafka' (no such file), 
 '/Users/<your user>/<your project>/bin/Debug/net6.0/liblibrdkafka' (no such file), 
 '/usr/local/lib/liblibrdkafka' (no such file), 
 '/usr/lib/liblibrdkafka' (no such file)

So we concluded that the actual package is missing from the right place.

So as a workaround you can do the following:

1. run command brew install librdkafka

2. copy all files from /opt/homebrew/Cellar/librdkafka/1.8.2/lib to one of the suggested location in the error (We choose the directory of the project ‘/Users/(your user)/(your project)’)

3. mark all added files as “Content” and “Copy always” in your IDE so they get transferred to the /bin/Debug folder on build (if you choose one of the other locations this is not needed)

4. after build everything should work without an error

PS: We tried setting the working directory of librdkafka with the line Library.Load("/opt/homebrew/Cellar/librdkafka/1.8.2/lib") but it didn’t work.