Support fields that are sometimes arrays

Using spark-1.6.2-bin-hadoop2.6, ES-Hadoop 2.4.0 and pyspark.

Somewhere in my dataset there are fields that are normally arrays, but occasionally aren’t - see below for an example. In real life, I think one offender is the logstash geoip filter generating the location field which is normally an array, but in some error cases isn’t.

If I set es_read_field_as_array_include I get scala.MatchError: jim, if I don’t set it I get scala.MatchError: Buffer(tom, peter). Therefore I have no way of reading the field, and can only process the index if I blacklist the offending fields.

Also the stacktrace doesn’t say what field it’s having the issue with, which makes debugging very hard.

curl -XPOST localhost:9200/sparktest/spark -d '{"friends":"jim"}'
curl -XPOST localhost:9200/sparktest/spark -d '{"friends":["tom","peter"]}'
./bin/pyspark --driver-class-path=../elasticsearch-hadoop-2.4.0/dist/elasticsearch-hadoop-2.4.0.jar
>>> df = sqlContext.read.format("org.elasticsearch.spark.sql").options(es_read_field_as_array_include="friends").load("sparktest/spark")
>>> df.show()

[aalkilani]

Any updates on this?

[CQUTWangHong]

When I print dataframe’s schema df.printSchema() It only show

root
 |-- string_field: string (nullable = false)

No display array_field that means “es.read.field.as.array.include” -> “array_field” doesn’t work