Fails to load ParquetSharpNative.so on CentOS/RedHat

ParquetSharp fails to load its native dependencies on certain Linux distributions. The issue probably comes from a wrong nuget packaging configuration, as ParquetSharpNative.so is not copied to the bin folder like it should.

Stacktrace:

Error Message:
   System.TypeInitializationException : The type initializer for 'ParquetSharp.LogicalTypeFactory' threw an exception.
  ----> System.DllNotFoundException : Unable to load shared library 'ParquetSharpNative' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libParquetSharpNative: cannot open shared object file: No such file or directory
  Stack Trace:
     at ParquetSharp.ParquetFileWriter..ctor(String path, Column[] columns, Compression compression, IReadOnlyDictionary`2 keyValueMetadata)
   at RQ.ParquetSharp.ParquetFileBuilder.WriteToParquet(String filename) in /Parquet/RQ.ParquetSharp/ParquetFileBuilder.cs:line 33
   at RQ.ParquetSharp.Test.TestParquetFileBuilder.TestParquetFileRoundTrip() in /Parquet/RQ.ParquetSharp.Test/TestParquetFileBuilder.cs:line 44
--DllNotFoundException
   at ParquetSharp.LogicalType.LogicalType_None(IntPtr& logicalType)
   at ParquetSharp.ExceptionInfo.Return[TValue](GetAction`1 getter)
   at ParquetSharp.LogicalType.None()
   at ParquetSharp.LogicalTypeFactory..cctor()

Affected Linux distributions:

• CentOS 7, 7.9.2009
• RedHat Enterprise 7.9

Affected ParquetSharp versions:

• Latest 7.0.0 Beta 1 on net5.0 (and older versions).

Debian 10 can load the native binary just fine.

[dgl]

Using a newer library on CentOS 7, or bringing one in via static linking is likely going to be troublesome, as this is being loaded as a library into the dotnet runtime, which is already using the libstdc++ and so on shipped with the OS.

Just so we’re talking about the same thing the ABI versions that can’t be found on CentOS 7 can be found by running ldd from CentOS within Docker, like so:

$ docker run -v $HOME:$HOME centos:7 ldd ~/ParquetSharpNative.so       
/home/dgl/ParquetSharpNative.so: /lib64/libgcc_s.so.1: version `GCC_7.0.0' not found (required by /home/dgl/ParquetSharpNative.so)
/home/dgl/ParquetSharpNative.so: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/dgl/ParquetSharpNative.so)
/home/dgl/ParquetSharpNative.so: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /home/dgl/ParquetSharpNative.so)
/home/dgl/ParquetSharpNative.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /home/dgl/ParquetSharpNative.so)
/home/dgl/ParquetSharpNative.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /home/dgl/ParquetSharpNative.so)
/home/dgl/ParquetSharpNative.so: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/dgl/ParquetSharpNative.so)
/home/dgl/ParquetSharpNative.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/dgl/ParquetSharpNative.so)
/home/dgl/ParquetSharpNative.so: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/dgl/ParquetSharpNative.so)
	linux-vdso.so.1 =>  (0x00007ffc811c3000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f6e5ad4e000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6e5ab32000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f6e5a82a000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f6e5a528000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6e5a312000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f6e59f44000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f6e5cb47000)

This shows the relevant ABI versions that can’t be found on CentOS 7. CXXABI is possible to solve via -fabi-version (https://gcc.gnu.org/onlinedocs/gcc/C_002b_002b-Dialect-Options.html), but GLIBC and so on are harder. There’s a nice overview of the problem at https://github.com/phusion/holy-build-box#problem-introduction

My suggestion would be to build on CentOS 7 (either via the above linked image, or centos:7 directly) if that’s the lowest target (as CentOS 7 is still within security support, rather than old versions of Ubuntu).

[mfkl]

Fixed in https://github.com/G-Research/ParquetSharp/releases/tag/7.0.0