[FEATURE] Example code for stream.pipeline()

Parsing or Formatting?

• Formatting
• Parsing

Is your feature request related to a problem? Please describe.

I have been dealing with an issue where end() is called before all the data is processed, so I am looking at using the stream.pipeline functionality, in NodeJS 12.

Describe the solution you’d like

It could be useful to provide an example of how to use fast-csv with the new stream.pipeline. Also including an indication of how to be sure that the stream has finished being processed before finishing, especially when using async/await.

Describe alternatives you’ve considered A clear and concise description of any alternative solutions or features you’ve considered.

Additional context Add any other context or screenshots about the feature request here.

[datiecher]

Thanks a lot, @doug-martin! That’s a more elegant and readable implementation of the solution I arrived at after digging deeper into the stream documentation yesterday.

[doug-martin]

@datiecher Yeah the node streams API isnt super straight forward. I would create a custom stream and pipe into it for this type of use case.

Take a look at https://nodejs.org/api/stream.html#stream_implementing_a_transform_stream it goes over the basics on implementing a stream, depending on what you need to do

Here is a quick example of how you could do it.

const { EOL } = require('os');

const csv = require('fast-csv');
const { Transform } = require('stream');


class ProductTransformStream extends Transform {
    _write(row, encoding, callback) {
        this.processRow(row).then((product) => {
            console.log(`DONE PROCESSING ROW ${JSON.stringify(row)}`);
            // push product if you want to use it it later streams.
            this.push(product);
            callback();
        }).catch(e => callback(e));
    }

    async processRow(row) {
        const product = await this.findProduct(row);

        if (product) {
            await this.updateProduct(product, row);
        } else {
            await this.createProduct(row);
        }

        await this.extraStuffForEachRow();
        return product;
    }

    async findProduct(row) {
        console.log(`Finding Product ${row.ID}`);
        return Promise.resolve({
            id: row.ID,
            name: row.name,
            price: row.price,
        });
    }

    async updateProduct(product) {
        console.log(`Updating Product ${product.id}`);
        return Promise.resolve(product);
    }

    async extraStuffForEachRow(row) {
        return Promise.resolve(row);
    }
}

const CSV_CONTENT = [
    'ID,name,price',
    `1,Cool Product,$100.00`,
    `2,Lame Product,$100.00`,
].join(EOL);

csv.parseString(CSV_CONTENT, { headers: true })
    .on('error', (e) => console.error(e.stack))
    .pipe(new ProductTransformStream({ objectMode: true }))
    .on('end', () => console.log('DONE!'))
    .on('error', (e) => console.error(e.stack));

If you are dealing with thousands of lines you may want to put some batching logic in there so you arent making each request individually.