chunkSize do not have any effect, defaults to around 65KB

Trying to set chunkSize to 50Kb but no matter what I set it seems to read round 65Kb chunk. Have tried all the 3 settings individually, but do not have any effect on chunk size (number of lines read from csv on each chunk call back remains the same)

options.chunkSize = 40000

Papa.RemoteChunkSize = 40000;

Papa.LocalChunkSize = 40000;

Even after setting options.chunkSize = null, Papa parses in multiple chunks

Please help …

[akash-rajput]

Any updates on this?

[WarrenWilkinson]

I had this issue using fs.createReadStream to create the file. It appears that there is a buffer inside the stream that’s about 10 MB. So it’s not PapaParse’s fault.

If this is your issue, you can pass parameters to fs.createReadStream to let it buffer more.
Something like this snippet should get you started…

Papa.LocalChunkSize =  Papa.LocalChunkSize * 10;
const file = fs.createReadStream(dataPath, { highWaterMark: Papa.LocalChunkSize });