Managing the Spark Session

In the spark-fast-tests README, we encourage users to wrap the Spark Session in a trait and mix in the trait to test classes that need access to the Spark Session.

import org.apache.spark.sql.SparkSession

trait SparkSessionTestWrapper {

  lazy val spark: SparkSession = {
    SparkSession.builder().master("local").appName("spark session").getOrCreate()
  }

}

The spark-testing-base library uses the EvilSessionTools approach to extract the SQL context.

I don’t think the testing framework should have any knowledge or control over the Spark Session. The Spark Session management should take place in the application and the test framework should simply provide tools that help with assertions.

@snithish @eclosson - I would like your feedback on this intentional design decision. Thanks!

[MrPowers]

@eclosson @snithish - I created an issue in the spark-testing-base repo: https://github.com/holdenk/spark-testing-base/issues/186 Feel free to chime in on the issue if you’d like to add anything.

[MrPowers]

@eclosson @snithish - Thanks for the feedback. I’m going to open an issue in the spark-testing-base library and see if they would consider removing the SparkSession management from DataFrameSuiteBase. I’d like the good ideas we develop in this testing framework to get reflected in the spark-testing-base repo as well since it’s more popular.