Normalizer is not applied on fields

NEST/Elasticsearch.Net version: 7.9.0 Elasticsearch version: 7.6.1 (Docker) Description of the problem including expected versus actual behavior: Hi, I basically need to query on text fields using a case insensitive Prefix to get all records having their productSearchName field “starting with” something.

Example of data :

productSearchName= "12 tons of ..."
productSearchName= "12 Tons of ..."
productSearchName= "12 TONS of ..."

I would like to get all these three records using a Prefix query, like

GET catalog/_search
{
  "_source": ["id", "productSearchName"], 
  "query": 
  {
    "prefix": {
      "productSearchName.Keyword": {
        "value": "12 tons"
      }
    }
  }
}

When defining a Normalizer (as I understood it’s the correct approach, is it ?) and telling my field should use it, the field is not mapped as expected, it’s a “basic” text fields on Keyword

        "productSearchName": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }

My query against the field indeed does not retrieve all records as expected. I indeed reindexed my whole documents on a dedicated index with the provided mapping.

Steps to reproduce: Here is my IndexDescriptor

            c => c
            .Settings(s => s.Analysis(a => a
			.Normalizers(n => n.Custom("case_insensitive", c => c.Filters("lowercase")))
            ))
            .Map<Catalog>(m => m
                .AutoMap()
                .Properties(p => p
                    .Keyword(st => st
                        .Name(n => n.ProductSearchName)
                            .Normalizer("case_insensitive")
                            )))

Expected behavior I expect the ProductSearchName field to use the provided normalizer, and then being able to query in a case insensitive way using Prefix instruction, and then get all my 3 example records.

[russcam]

No worries, @NicolasReyDotNet. You can also check to see if the index exists before attempting to create, with client.Indices.Exist()

[russcam]

Hi @NicolasReyDotNet,

Creating an index with a normalizer specified in analysis and applying to a type field works as expected. Here’s an example that’ll also log out the request and response

private static void Main()
{
	var defaultIndex = "documents";
	var pool = new SingleNodeConnectionPool(new Uri($"https://localhost:9200"));
	
	var settings = new ConnectionSettings(pool)
	    .DefaultIndex(defaultIndex)
	    .DisableDirectStreaming()
		.PrettyJson()
		.OnRequestCompleted(callDetails =>
	    {
	        if (callDetails.RequestBodyInBytes != null)
	        {
	            var json = JObject.Parse(Encoding.UTF8.GetString(callDetails.RequestBodyInBytes));
	            
	            Console.WriteLine(
	                $"{callDetails.HttpMethod} {callDetails.Uri} \n" +
	                $"{json.ToString(Newtonsoft.Json.Formatting.Indented)}");
	        }
	        else
	        {
	            Console.WriteLine($"{callDetails.HttpMethod} {callDetails.Uri}");
	        }
	
	        Console.WriteLine();
	
	        if (callDetails.ResponseBodyInBytes != null)
	        {
	            Console.WriteLine($"Status: {callDetails.HttpStatusCode}\n" +
	                     $"{Encoding.UTF8.GetString(callDetails.ResponseBodyInBytes)}\n" +
	                     $"{new string('-', 30)}\n");
	        }
	        else
	        {
	            Console.WriteLine($"Status: {callDetails.HttpStatusCode}\n" +
	                     $"{new string('-', 30)}\n");
	        }
	    });
	    
	var client = new ElasticClient(settings);
	
	var createIndexResponse = client.Indices.Create("foo", c => c
		.Settings(s => s.Analysis(a => a
				.Normalizers(n => n.Custom("case_insensitive", c => c.Filters("lowercase")))
				))
		.Map<Catalog>(m => m
			.AutoMap()
			.Properties(p => p
				.Keyword(st => st
					.Name(n => n.ProductSearchName)
						.Normalizer("case_insensitive")
		)))
	);

}

public class Catalog
{
	public string ProductSearchName {get;set;}
}

which yields

PUT https://localhost:9200/foo?pretty=true 
{
  "mappings": {
    "properties": {
      "productSearchName": {
        "normalizer": "case_insensitive",
        "type": "keyword"
      }
    }
  },
  "settings": {
    "analysis": {
      "normalizer": {
        "case_insensitive": {
          "filter": [
            "lowercase"
          ],
          "type": "custom"
        }
      }
    }
  }
}

It looks like a document may have been indexed into the index before the explicit index creation and mapping was applied; The productSearchName is mapped with the default inference mapping for a string that would be generated.