Example usage of ParquetWriter.openStream?

I saw that @asmuth implemented it here: https://github.com/ironSource/parquetjs/blob/master/lib/writer.js#L52 But I’m wondering if someone has a simple example of how to use this.

I’ve got it mostly working but I had to implement a close method on a PassThrough stream, I feel might be doing it incorrectly.

I’d be happy to submit a PR with some nice doc as well once I have it all figured out since I imagine this is a common enough use case.

[ali-habibzadeh]

The easier way I have found is not using ParquetWriter.openStream but via an undocumented ParquetTransformer class. This class extends node’s Transform stream and can be used inside a pipe as a step.

import { createReadStream, createWriteStream } from "fs";
import { ParquetSchema, ParquetTransformer } from "parquetjs";
// stream-json has some nice streaming tools for working with JSON
import * as StreamArray from "stream-json/streamers/StreamArray";

const reader = createReadStream("data-json.json"); // contains JSON Array
const destination = createWriteStream("countries.parquet");

const schema = new ParquetSchema({
  value: {
    fields: {
      Country: { type: "UTF8" },
      Indicator: { type: "UTF8" },
      Value: { type: "FLOAT" },
      Year: { type: "INT64" }
    }
  }
});

reader
  .pipe(StreamArray.withParser())
  .pipe(new ParquetTransformer(schema))
  .pipe(destination);

[dogenius01]

Similar issue. How to write parquet file using stream?