Performance issue

Hi,

I tested the ParquetSource on a parquet file of about 5M records, split in 144 partitions (files) on an 8 CPU server in the default way:

    ParquetSource(new Path(modelPath))
      .toFrame(16)
      .toList
      .map(convertRow)

After about 10 min the process got stuck and I had to kill it. All cpu’s were at 100% for the whole duration

I then tested with a slightly modified codebase:

    ParquetSource(new Path(modelPath))
      .parts
      .par
      .flatMap { part =>
        part.data
          .map(convertRow)
          .toList.toBlocking.single
      }

And that finished the job in 30 sec. Even without the .par line it still only takes 2 min (running on a single cpu)

Any idea why the two implementations give so different results?

[sksamuel]

Yes you can do that, or the Frame API will allow you to multi-thread the code in the next release as well. The reason for the List[Row] is that it avoids a lot of the contention on the locks.

On 7 December 2016 at 09:58, Niek Bartholomeus [e-mail] wrote:

Looks good!

I currently don’t have the full dataset of 5M words available but I tested with roughly a fifth of it. With the previous version the load took 15s and with the latest version it only takes 10s!

I will let you know the figures once I have my 5M dataset back.

One note: as I want to run parallel calculations based on the loaded dataset, I prefer to keep it split up in a ParMap. The parts api allows me to do this so I’m not using the ``toFrame``` api. I noticed however that the latest version is less straightforward to collect all rows per part:

ParquetSource(new Path(wordVariationPath))
  .parts
  .par
  .flatMap { part =>
    var rowsTotal = List[Row]()
    part.iterator.foreach { rows =>
      rowsTotal = rowsTotal ++ rows
    }
  }

Is this the best way to proceed?

Thanks!

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/sksamuel/eel-sdk/issues/180#issuecomment-265404885, or mute the thread https://github.com/notifications/unsubscribe-auth/AAtZGgT7C9eyM2ppEvlit6F2mdun5rvYks5rFoNBgaJpZM4K7dpH .

[tolomaus]

Yes indeed, the performance is fine now. Ok I will see if I can reproduce the conversion error with a minimal scenario.