🈁💌💗 Post your use cases / software / screenshots here

Hi everyone! This is an open thread for posting examples of the way you are using Parquet.Net. I’m looking for real world usage!

For example:

I use Parquet.Net to store and query my collection of memes. It’s very fast and efficient, and it supports all kinds of data types, like strings, integers, Booleans, and even emojis! 😂

Of course it’s a joke, but you get the idea!

How do you use Parquet.Net? Share your examples below, and don’t forget to check you are not violating your employer’s privacy policy, intellectual rights and so on. You don’t want to end up like Bob, who accidentally leaked the company’s secret formula for their best-selling product. He thought he was just sharing a harmless photo of his lunch, but he didn’t notice the paper on his desk with the confidential information. Now he is looking for a new job and a new place to eat.

[immitev]

We use the library for implementing SSIS+ components for reading/writing Parquet files https://cozyroc.com/ssis/parquet .

Well, the initial development is based on an older version of the library, due to requiring .NET Framework 4.0 support (did some backporting). Luckily the upcoming version of our component suite can target .NET Framework 4.6.1, so that we are transitioning to the 3.10.0 version.

[sbrickey]

we use it as a replacement for Azure Data Factory, which saves us about $10k/month

our design is simple: table to store source/destinations and some tracking flags. Process will check the table, identify which records are “ready”, transfers data, then updates the records to indicate their completion… we use this to keep cloud data “current” from the source system(s)

ADF cost was split about half between “bytes transferred” and “API calls” (interactions with the table above)

we replaced it with custom code running on a B-series VM, which costs around $130/month

the custom code is simple: generate local parquet file from source, upload to storage, execute API call into databricks to merge parquet file into existing data.