gms report ES all shards failed when ES index mapping upgrade

Due to biz needs, the dataset-index mapping definition has been adjusted. But after I adjust the definition, the gms service will report an exception.

Reproduce step:

curl -X POST \
  http://10.190.81.17:9200/datasetdocument/_search \
  -H 'Content-Type: application/json' \
  -d '{
	"query": {
		"query_string": {
			"query": "description:维度"
		}
	}
}'

curl -sS -XPUT http://10.190.81.17:9200/datasetdocumentv2 -H 'Content-Type: Application/JSON' --data @dataset-index-config.json | jq .

curl -X POST \
  http://10.190.81.17:9200/_reindex \
  -H 'Content-Type: application/json' \
  -d '{
  "source": {
    "index": "datasetdocumentv2"
  },
  "dest": {
    "index": "datasetdocument"
  }
}'

curl -X POST \
  http://10.190.81.17:9200/_aliases \
  -H 'Content-Type: application/json' \
  -d '{
  "actions": [
    {
      "add": {
        "index": "datasetdocumentv2",
        "alias": "datasetdocument"
      }
    },
    {
      "remove_index": {
        "index": "datasetdocument"
      }
    }
  ]
}'

Exception:

04:26:53.021 [qtp626202354-16] ERROR c.l.metadata.dao.browse.ESBrowseDAO - Browse query failed: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
04:26:56.098 [qtp626202354-20] ERROR c.l.metadata.dao.search.ESSearchDAO - Auto complete query failed:Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
04:26:56.233 [qtp626202354-13] ERROR c.l.metadata.dao.search.ESSearchDAO - Search query failed:Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
...

[cobolbaby]

@camelliazhang the dataset BACK_FILL is awesome.

[camelliazhang]

@cobolbaby Reindex API does not work well for your case. If you have made changes on index mapping, you will need to bootstrap your new index (let’s call it new_index). How many datasets do you have in total? You may need to either replay all the MCEs or call dataset BACK_FILL to fill your new index. After the new_index is completely populated, you can then do alias switch with current live index.

Internally we are actively working on the improvement for the backfill process, will provide documentation about it too.