[Read] Read Decimal LogicalType

Hello,

Do you have any example to read a decimal column in a parquet file ?

I am stuck on error “System.NotSupportedException: only 16 bytes of decimal length is supported”

PhysicalType = FixedLenByteArray

i dont find workaround since it breaks whenever i call Column.LogicalReader<T>

Maybe is there a method to get raw data like Column.PhysicalReader<T> ?

Br Nick

What i tried so far:

case LogicalTypeEnum.Decimal:
      return column.LogicalReader<decimal?>().ReadAll(numRows)
          .Select(l => (object)l)
          .ToArray();

[adamreeve]

To handle nullable values you need to use the ReadBatch override that takes defLevels and repLevels. In the case of a column of scalar nullable values (not nested in an array or other struct) then repLevels can be ignored. Eg. something like this should work:

var typeLength = column.ColumnDescriptor.TypeLength;
var definitionLevel = column.ColumnDescriptor.MaxDefinitionLevel;
var values = new FixedLenByteArray[numRows];
var defLevels = new short[numRows];
var bytes = new byte[typeLength];

var decimalValues = new List<decimal?>();

// create byte vector values
var coefs = Enumerable.Range(0, column.ColumnDescriptor.TypeLength)
    .Reverse()
    .Select(iB => (decimal)Math.Pow(256, iB))
    .ToArray();

long totalRowsRead = 0;
while (totalRowsRead < numRows)
{
    var rowsRead = _column.ReadBatch(numRows, defLevels, null, values, out var valuesRead);
    var valueIdx = 0;
    for (var i = 0; i < rowsRead; ++i)
    {
        if (defLevels[i] == definitionLevel)
        {
            Marshal.Copy(values[valueIdx].Pointer, bytes, 0, typeLength);
            // Parse the string as a decimal value
            decimalValues.Add(ConvertFixedLengthByteArrayToDecimal(bytes, column.ColumnDescriptor.TypeScale, coefs));
            valueIdx++;
        }
        else
        {
            decimalValues.Add(null);
        }
    }

    totalRowsRead += rowsRead;
}

[adamreeve]

Hi @Platob, which version of ParquetSharp are you using, and do you know the byte width of the decimal data you are trying to read? If your data uses a physical type of int32 or int64 then support for reading this has been added in ParquetSharp 10.0.1-beta1 (#315).

Otherwise we only currently support reading fixed-length byte array based decimal data that uses 16 bytes, but it is possible to read the raw byte values. See https://github.com/G-Research/ParquetSharp/discussions/317#discussioncomment-4210430 for an example that just prints a hex representation of the byte values.