Position for 'params.expo_cnt' not found in row; typically this is caused by a mapping inconsistency

What kind an issue is this?

• Bug report. If you’ve found a bug, please provide a code snippet or test to reproduce it below.
  The easier it is to track down the bug, the faster it is solved.
• Feature Request. Start by telling us what problem you’re trying to solve.
  Often a solution already exists! Don’t send pull requests to implement new features without first getting our support. Sometimes we leave features out on purpose to keep the project small.

Issue description

org.elasticsearch.hadoop.rest.EsHadoopParsingException: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Position for ‘params.expo_cnt’ not found in row; typically this is caused by a mapping inconsistency at org.elasticsearch.hadoop.serialization.ScrollReader.readHit(ScrollReader.java:514) at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:292) at org.elasticsearch.hadoop.serialization.ScrollReader.read(ScrollReader.java:262) at org.elasticsearch.hadoop.rest.RestRepository.scroll(RestRepository.java:313) at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(ScrollQuery.java:93) Description when read data from es, I got the above error. mapping file:

"mappings": {
  "all": {
    "properties": {
      "ftime": {
        "type": "long"
      },
      "metadata": {
        "properties": {
          "fduration": {
            "type": "long"
          }
        }
      },
      "gvid": {
        "type": "text",
        "fields": {
          "keyword": {
            "ignore_above": 256,
            "type": "keyword"
          }
        }
      },
      "params": {
        "properties": {
          "share_cnt": {
            "type": "long"
          },
          "valid_play_cnt": {
            "type": "long"
          },
          "expo_cnt": {
            "type": "long"
          },
          "strict_full_play_cnt": {
            "type": "long"
          },
          "follow_cnt": {
            "type": "long"
          },
          "notlike_cnt": {
            "type": "long"
          },
          "full_play_cnt": {
            "type": "long"
          },
          "loose_full_play_cnt": {
            "type": "long"
          },
          "like_cnt": {
            "type": "long"
          },
          "play_cnt": {
            "type": "long"
          },
          "playtime_sum": {
            "type": "long"
          }
        }
      }
    }
  }
},

Steps to reproduce

Code: spark = SparkSession .builder() .config(“es.nodes”, Configuration.es_video_nodes) .config(“es_port”, Configuration.es_port) .enableHiveSupport() .getOrCreate() val df = EsSparkSQL.esDF(spark, Configuration.index) df.show(10)

Test/code snippet

Strack trace:

Stack trace goes here

Version Info

OS: :
JVM :
Hadoop/Spark: 2.3.1 ES-Hadoop : 7.0 ES : 7.0

Feature description

[pnuzyf]

is there any progress?

[jbaiera]

“_source” : { “ftime” : 20200308161000, “params.expo_cnt” : 0, “params.strict_full_play_cnt” : 0, “params.valid_play_cnt” : 2, “params.share_cnt” : 0, “params.loose_full_play_cnt” : 2, “params.notlike_cnt” : 0, “params.playtime_sum” : 190, “params.like_cnt” : 0, “params.follow_cnt” : 0, “gvid” : “r01854c8m1n”, “metadata.fduration” : 1000, “params.play_cnt” : 2, “params.full_play_cnt” : 0 }, “_source” : { “ftime” : 20200308160500, “params.expo_cnt” : 0, “params.strict_full_play_cnt” : 0, “params.valid_play_cnt” : 1, “params.share_cnt” : 0, “params.loose_full_play_cnt” : 1, “params.notlike_cnt” : 0, “params.playtime_sum” : 88, “params.like_cnt” : 0, “params.follow_cnt” : 0, “gvid” : “r01854c8m1n”, “metadata.fduration” : 1000, “params.play_cnt” : 1, “params.full_play_cnt” : 0 }

Sorry for the long response time - Are you seeing this issue with a normalized data schema (no dotted field names)? Right now ES-Hadoop does not support dotted fields names despite ES’s acceptance of them (#853) due to a number of issues pertaining to how mappings are discovered and how to maintain symmetric serialization of data.