duplicates when parsing - max workerNum ?

I’m having duplicate objects in m parsing results. I’m parsing large CSV files (using fromFile and the on('json') event) using workers.

When i set workerNum:4 everything seems to be ok (as far as i can tell), but if i use more i get duplicates. For example: when i use workerNum:8 i get double objects in my parsed result. When i use workerNum:12 i get triple duplicates.

any idea why? is there a limit of 4 workers?

Note: my machine has 48 vCPU

UPDATE: seems like 4 workers also produced duplicates

[Keyang]

Hi, Please checkout Asynchronously Process section here https://github.com/Keyang/node-csvtojson#asynchronouse-result-process

For your situation you can do:

csvToJson({
    maxRowLength: 65535
}).fromFile(csvFilePath)
.subscribe(async (json)=>{
   const storeResult=await db.insert(json)
})

I have temporarily removed support of workerNum in v2 as workers created too much overhead on inter process communication. The background is the at the same situation. If you are building a node.js based web service, it would probably be better to use built-in cluster feature to utilise multiple cores.

I will add in another way to support multiple cpu cores for parsing fromFile in future as seems divide file into multiple chunk is the only proper way to parse in parallel.

~Keyang

[Fire-Brand]

Great, thank you! Can’t wait for parallel parsing, not sure i would go for cluster right now, but i’ll definitely look into it!