Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[FEATURE] Example code for stream.pipeline()

See original GitHub issue

Parsing or Formatting?

  • Formatting
  • Parsing

Is your feature request related to a problem? Please describe.

I have been dealing with an issue where end() is called before all the data is processed, so I am looking at using the stream.pipeline functionality, in NodeJS 12.

Describe the solution you’d like

It could be useful to provide an example of how to use fast-csv with the new stream.pipeline. Also including an indication of how to be sure that the stream has finished being processed before finishing, especially when using async/await.

Describe alternatives you’ve considered A clear and concise description of any alternative solutions or features you’ve considered.

Additional context Add any other context or screenshots about the feature request here.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

datiechercommented, May 27, 2020

Thanks a lot, @doug-martin! That’s a more elegant and readable implementation of the solution I arrived at after digging deeper into the stream documentation yesterday.

doug-martincommented, May 27, 2020

@datiecher Yeah the node streams API isnt super straight forward. I would create a custom stream and pipe into it for this type of use case.

Take a look at it goes over the basics on implementing a stream, depending on what you need to do

Here is a quick example of how you could do it.

const { EOL } = require('os');

const csv = require('fast-csv');
const { Transform } = require('stream');

class ProductTransformStream extends Transform {
    _write(row, encoding, callback) {
        this.processRow(row).then((product) => {
            console.log(`DONE PROCESSING ROW ${JSON.stringify(row)}`);
            // push product if you want to use it it later streams.
        }).catch(e => callback(e));

    async processRow(row) {
        const product = await this.findProduct(row);

        if (product) {
            await this.updateProduct(product, row);
        } else {
            await this.createProduct(row);

        await this.extraStuffForEachRow();
        return product;

    async findProduct(row) {
        console.log(`Finding Product ${row.ID}`);
        return Promise.resolve({
            id: row.ID,
            price: row.price,

    async updateProduct(product) {
        console.log(`Updating Product ${}`);
        return Promise.resolve(product);

    async extraStuffForEachRow(row) {
        return Promise.resolve(row);

const CSV_CONTENT = [
    `1,Cool Product,$100.00`,
    `2,Lame Product,$100.00`,

csv.parseString(CSV_CONTENT, { headers: true })
    .on('error', (e) => console.error(e.stack))
    .pipe(new ProductTransformStream({ objectMode: true }))
    .on('end', () => console.log('DONE!'))
    .on('error', (e) => console.error(e.stack));

If you are dealing with thousands of lines you may want to put some batching logic in there so you arent making each request individually.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Node.js Stream.pipeline() Method - GeeksforGeeks
The stream.pipeline() method is a module method that is used to the pipe by linking the streams passing on errors and accurately cleaning...
Read more >
stream.pipeline JavaScript and Node.js code examples
Best JavaScript code snippets using stream.pipeline(Showing top 11 results ... const accumulator = new ArrAccumulator() pipeline( fibStream, accumulator, ...
Read more >
How to connect streams with pipeline? - Mario Kandut
Let's look at a code example. First we are going to create a sample file, then we are going to create a pipeline,...
Read more >
10 Examples of Stream API in Java 8 - count + filter + map + ...
Here is an example of counting how many elements are in the stream at any stage of pipeline processing using the count() method...
Read more >
Stream | Node.js v19.3.0 Documentation
Duplex : streams that are both Readable and Writable (for example, net.Socket ). ... A key goal of the stream API, particularly the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found