Support variable-size byte array encoding for decimals
See original GitHub issueVersion: Parquet.Net v3.9.1
Runtime Version: .Net 5.0
OS: Windows 10
Expected behavior
I would expect Parquet.Net to identify the DataTypeHandler appropriately for the column in question.
Actual behavior
Parquet.Net is throwing the aforementioned exception when I access parquetReader.Schema
.
From ParquetViewer, the column in question is:
"Schema": [
{
"Field_id": 0,
"Name": "CONTACT_ID",
"Type": "BYTE_ARRAY",
"Type_length": 0,
"LogicalType": null,
"Scale": 4,
"Precision": 23,
"Repetition_type": "OPTIONAL",
"Converted_type": "DECIMAL"
},
Steps to reproduce the behavior
- Step 1 - open a
ParquetReader
against the file. - Step 2 - access
ParquetReader.Schema
when one of the columns is as described above
Code snippet reproducing the behavior
using (Stream fileStream = System.IO.File.OpenRead(fileName))
{
using (var parquetReader = new ParquetReader(fileStream))
{
Schema schema = parquetReader.Schema;
}
}
I am unfortunately unable to share a copy of the file in question. If there is a tool that will allow me to extract a subset easily, I could likely share a file with just this column.
Issue Analytics
- State:
- Created a year ago
- Comments:11 (5 by maintainers)
Top Results From Across the Web
converting decimal to actual bytes - python
... bytes object Construct an immutable array of bytes from: - an iterable yielding integers in range(256) - a text string encoded using...
Read more >Working with Bytes
This guide will help you encode different types of data in as little bytes possible. The unprecedented range of the LoRaWAN technology we...
Read more >The length of datatype decimal is returned as 8 bytes ...
Instances of datatype Decimal occupy 16 bytes, as per documentation. Only that it doesn't return 16, but 8 bytes. Is this intentional, or...
Read more >Working with bytes in Dart - Suragch - Medium
The first example takes the decimal number 2020 , converts it to base-16 (i.e., hexadecimal), and makes sure the length is 4. Any...
Read more >Converting integer to byte string problem in python 3
I have a library function that is failing because I cannot seem to properly convert an integer to a proper byte string except...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Not ready yet )
Issue resolved in v4.3.3