add option to do a spark-submit with a SparkListener to gather events from Spark
See original GitHub issueI was at Emily Curtin’s Spark Summit Europe presentation today (which was very interesting). An attendee asked if Spark Bench gathered Spark executor metrics.
A SparkListener can be used to get benchmark data about how long was spent running tasks and how much data was shuffled (basically any data that can be seen in the Spark UI could be picked up and summarised).
https://spark.apache.org/docs/2.2.0/api/java/org/apache/spark/scheduler/SparkListener.html
spark-submit --conf spark.extraListeners=com.mycompany.MetricsListener
https://github.com/LucaCanali/sparkMeasure has a spark listener that gathers metrics.
https://github.com/groupon/sparklint also has one.
One possible design would be to
- run spark-submit with a SparkListener that outputs the event data (eg as CSV)
- run another spark job to summarise the event data and include the summary metrics with the other benchmark data
Another approach would be to run spark with spark.eventLog.enabled=true
(and spark.eventLog.dir
set) and parsing the json-lines output. https://github.com/groupon/sparklint also has code to summarise event logs to create metrics.
Issue Analytics
- State:
- Created 6 years ago
- Comments:5 (4 by maintainers)
I have a very early prototype at https://github.com/pjfanning/spark-bench/pull/2/files
Running
bin/spark-bench.sh examples/minimal-example.conf
on a distro with my change outputsThe aim is to gather more metrics with the listener and to include them with the other benchmarks. This would involve writing the metric data to a file and having spark-bench read that data and extending the benchmark data with these additional metrics.
a CSV file recording the task-durations of all tasks would be better.