Using es.index.read.missing.as.empty returns an empty result if one out of many indices is missing

If a user specifies multiple indices and one of them is not found - if the es.index.read.missing.as.empty property is set to true, then the entire connector returns an empty result. Instead, the connector should skip the missing index and continue on with processing the rest of the provided indices.

[alrutten]

similar issue. Fetching data using spark-sql throws and exception when an index does not exist, in stead of continuing with the next index when es.index.read.missing.as.empty is set to true.

ES options:

val elasticSearchOptions = Map(
		ConfigurationOptions.ES_NODES -> "node01, node02",
		ConfigurationOptions.ES_PORT -> esPort,
		ConfigurationOptions.ES_INDEX_AUTO_CREATE -> "yes",
		ConfigurationOptions.ES_RESOURCE_READ -> "graylog2_*",
		ConfigurationOptions.ES_INDEX_READ_MISSING_AS_EMPTY -> "true",
		"pushdown" -> "true",
		ConfigurationOptions.ES_HTTP_TIMEOUT -> "10m",
		ConfigurationOptions.ES_SCROLL_KEEPALIVE -> "60m",
		ConfigurationOptions.ES_SCROLL_SIZE -> "10000",
		ConfigurationOptions.ES_MAPPING_DATE_RICH_OBJECT -> "false",
		ConfigurationOptions.ES_MAX_DOCS_PER_PARTITION -> "1000000"
	)

‘’’

querying ES from spark:

val log: DataFrame = EsSparkSQL.esDF(session, elasticSearchOptions)

forcing evaluation: log.cache()

this inconsistently runs into org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: no such index

i assume this happens when we get unlucky and the index gets deleted between spark querying ES, and actually fetching the data. Note: this is unrelated to whether the queried data is on the deleted index or not (it never was).

[masseyke]

Yeah the pull request is at #1997.