RuntimeException: scala.Some is not a valid external type for schema of string
See original GitHub issueI’m trying to read an Excel file in Java using the spark-excel
library. It appears that the returned data is wrapped in scala.Some
type.
Here is how I invoke the library:
Dataset<Row> extractedExcel = sparkSession
.read()
.format("com.crealytics.spark.excel")
.option("inferSchema", "false")
.option("treatEmptyValuesAsNulls", "false")
.option("useHeader", "true")
// .schema(schema)
.load(absPath);
Then, if I make any call on the resulting dataset, say extractedExcel.show();
or try to write the dataset out, I run into an exception:
Caused by: java.lang.RuntimeException: scala.Some is not a valid external type for schema of string
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.StaticInvoke_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:324)
The error message suggests that the returned dataset wraps values in a scala.Some
. I don’t see how can the values be unwrapped and turned into plain strings.
Issue Analytics
- State:
- Created 4 years ago
- Reactions:1
- Comments:13
Top Results From Across the Web
Spark 3.0 scala.None$ is not a valid external type for schema ...
It worked by setting elasticsearch-hadoop property es.field.read.empty.as.null = no .option("es.field.read.empty.as.null", "no").
Read more >Spark job fails after upgrade from DSE 5.0 with error "is not a ...
RuntimeException : Error while encoding: java.lang.RuntimeException: scala.None$ is not a valid external type for schema of date ...
Read more >UTF8String is not a valid external type for schema of string
I'm having this error using the 3.0.0-alpha2 with spark 3.0.0-preview2, below is a minimal example to run in the spark-shell and the stacktrace, ......
Read more >Unable to read data from Elasticsearch with spark in Databricks.
Caused by: RuntimeException: scala.collection.convert.Wrappers$JListWrapper is not a valid external type for schema of string · _id · _index ...
Read more >tFileInputParquet randomly changing the order of the schema ...
RuntimeException : Error while encoding: java.lang.RuntimeException: java.sql.Timestamp is not a valid external type for schema of string.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I just did a release, should be available very soon (maybe already).
@EnverOsmanov Thanks for replying, the issue is solved after I ran my code with latest version of spark-excel(0.13.5)