question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Support variable-size byte array encoding for decimals

See original GitHub issue

Version: Parquet.Net v3.9.1

Runtime Version: .Net 5.0

OS: Windows 10

Expected behavior

I would expect Parquet.Net to identify the DataTypeHandler appropriately for the column in question.

Actual behavior

Parquet.Net is throwing the aforementioned exception when I access parquetReader.Schema.

From ParquetViewer, the column in question is:

  "Schema": [
    {
      "Field_id": 0,
      "Name": "CONTACT_ID",
      "Type": "BYTE_ARRAY",
      "Type_length": 0,
      "LogicalType": null,
      "Scale": 4,
      "Precision": 23,
      "Repetition_type": "OPTIONAL",
      "Converted_type": "DECIMAL"
    },

Steps to reproduce the behavior

  1. Step 1 - open a ParquetReader against the file.
  2. Step 2 - access ParquetReader.Schema when one of the columns is as described above

Code snippet reproducing the behavior

using (Stream fileStream = System.IO.File.OpenRead(fileName))
{
   using (var parquetReader = new ParquetReader(fileStream))
   {
      Schema schema = parquetReader.Schema;
   }
}

I am unfortunately unable to share a copy of the file in question. If there is a tool that will allow me to extract a subset easily, I could likely share a file with just this column.

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:11 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
aloneguidcommented, Jan 11, 2023

Not ready yet )

0reactions
jchristncommented, Jan 21, 2023

Issue resolved in v4.3.3

Read more comments on GitHub >

github_iconTop Results From Across the Web

converting decimal to actual bytes - python
... bytes object Construct an immutable array of bytes from: - an iterable yielding integers in range(256) - a text string encoded using...
Read more >
Working with Bytes
This guide will help you encode different types of data in as little bytes possible. The unprecedented range of the LoRaWAN technology we...
Read more >
The length of datatype decimal is returned as 8 bytes ...
Instances of datatype Decimal occupy 16 bytes, as per documentation. Only that it doesn't return 16, but 8 bytes. Is this intentional, or...
Read more >
Working with bytes in Dart - Suragch - Medium
The first example takes the decimal number 2020 , converts it to base-16 (i.e., hexadecimal), and makes sure the length is 4. Any...
Read more >
Converting integer to byte string problem in python 3
I have a library function that is failing because I cannot seem to properly convert an integer to a proper byte string except...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found