ExtractEntityMetadata crashes on specific entity

Version

“Microsoft.Bot.Builder” Version=“4.9.3”

Describe the bug

An specific german entity, namely “Sicherheitsdienstmitarbeiterinnen” throws an exception in LuisUtil:ExtractEntityMetadata() when calling .RecognizeAsync() on the Dispatch Model.

To Reproduce

I have created a minimal .lu example to generate a LUIS Language Model to reproduce the error.

## Demo
- {@test}

@ list test = 
	- Sicherheitsdienst :
		- Sicherheitsdienstmitarbeiterinnen
		- Sicherheitsdienst mitarbeiterinnen

## None
- test

And here the matching DispatchLuis.cs

// <auto-generated>
// Code generated by luis:generate:cs
// Tool github: https://github.com/microsoft/botframework-cli
// Changes may cause incorrect behavior and will be lost if the code is
// regenerated.
// </auto-generated>
using Newtonsoft.Json;
using Newtonsoft.Json.Serialization;
using System;
using System.Collections.Generic;
using Microsoft.Bot.Builder;
using Microsoft.Bot.Builder.AI.Luis;
namespace Luis
{
    public partial class DispatchLuis: IRecognizerConvert
    {
        [JsonProperty("text")]
        public string Text;

        [JsonProperty("alteredText")]
        public string AlteredText;

        public enum Intent {
            l_General,
            None
        };
        [JsonProperty("intents")]
        public Dictionary<Intent, IntentScore> Intents;

        public class _Entities
        {
            // Lists
            public string[][] test;


            // Instance
            public class _Instance
            {
                public InstanceData[] test;
            }
            [JsonProperty("$instance")]
            public _Instance _instance;
        }
        [JsonProperty("entities")]
        public _Entities Entities;

        [JsonExtensionData(ReadData = true, WriteData = true)]
        public IDictionary<string, object> Properties {get; set; }

        public void Convert(dynamic result)
        {
            var app = JsonConvert.DeserializeObject<DispatchLuis>(
                JsonConvert.SerializeObject(
                    result,
                    new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore, Error = OnError }
                )
            );
            Text = app.Text;
            AlteredText = app.AlteredText;
            Intents = app.Intents;
            Entities = app.Entities;
            Properties = app.Properties;
        }

        private static void OnError(object sender, ErrorEventArgs args)
        {
            // If needed, put your custom error logic here
            Console.WriteLine(args.ErrorContext.Error.Message);
            args.ErrorContext.Handled = true;
        }

        public (Intent intent, double score) TopIntent()
        {
            Intent maxIntent = Intent.None;
            var max = 0.0;
            foreach (var entry in Intents)
            {
                if (entry.Value.Score > max)
                {
                    maxIntent = entry.Key;
                    max = entry.Value.Score.Value;
                }
            }
            return (maxIntent, max);
        }
    }
}

Steps to reproduce the behavior:

1. Generate and train the LUIS Language Model + Dispatch Model for the given .lu snippet.
2. Call var dispatchResult = await localizedServices.DispatchService.RecognizeAsync<DispatchLuis>(innerDc.Context, cancellationToken); in the MainDialog of your Bot.
3. Get the following error:

Index and length must refer to a location within the string. (Parameter ‘length’)

" at System.String.Substring(Int32 startIndex, Int32 length)\n at Microsoft.Bot.Builder.AI.Luis.LuisUtil.ExtractEntityMetadata(EntityModel entity, String utterance)\n at Microsoft.Bot.Builder.AI.Luis.LuisUtil.ExtractEntitiesAndMetadata(IList1 entities, IList1 compositeEntities, Boolean verbose, String utterance)\n at Microsoft.Bot.Builder.AI.Luis.LuisRecognizerOptionsV2.RecognizeInternalAsync(ITurnContext turnContext, HttpClient httpClient, CancellationToken cancellationToken)\n at Microsoft.Bot.Builder.AI.Luis.LuisRecognizer.RecognizeInternalAsync(ITurnContext turnContext, LuisRecognizerOptions predictionOptions, Dictionary2 telemetryProperties, Dictionary2 telemetryMetrics, CancellationToken cancellationToken)\n at Microsoft.Bot.Builder.AI.Luis.LuisRecognizer.RecognizeAsync(ITurnContext turnContext, CancellationToken cancellationToken)\n at WienerLinienBot.Dialogs.MainDialog.OnContinueDialogAsync(DialogContext innerDc, CancellationToken cancellationToken) in /Users/hanneshasenauer/Source/WienerLinienBot/WienerLinienBot/Dialogs/MainDialog.cs:line 140\n at Microsoft.Bot.Builder.Dialogs.ComponentDialog.ContinueDialogAsync(DialogContext outerDc, CancellationToken cancellationToken)\n at Microsoft.Bot.Builder.Dialogs.DialogContext.ContinueDialogAsync(CancellationToken cancellationToken)\n at Microsoft.Bot.Builder.Dialogs.DialogExtensions.RunAsync(Dialog dialog, ITurnContext turnContext, IStatePropertyAccessor1 accessor, CancellationToken cancellationToken)\n at Microsoft.Bot.Builder.ActivityHandler.OnTurnAsync(ITurnContext turnContext, CancellationToken cancellationToken)\n at WienerLinienBot.Bots.DefaultActivityHandler1.OnTurnAsync(ITurnContext turnContext, CancellationToken cancellationToken) in /Users/hanneshasenauer/Source/WienerLinienBot/WienerLinienBot/Bots/DefaultActivityHandler.cs:line 35\n at Microsoft.Bot.Solutions.Middleware.EventDebuggerMiddleware.OnTurnAsync(ITurnContext turnContext, NextDelegate next, CancellationToken cancellationToken)\n at Microsoft.Bot.Solutions.Middleware.SetLocaleMiddleware.OnTurnAsync(ITurnContext context, NextDelegate next, CancellationToken cancellationToken)\n at Microsoft.Bot.Builder.TranscriptLoggerMiddleware.OnTurnAsync(ITurnContext turnContext, NextDelegate nextTurn, CancellationToken cancellationToken)\n at Microsoft.Bot.Builder.TelemetryLoggerMiddleware.OnTurnAsync(ITurnContext context, NextDelegate nextTurn, CancellationToken cancellationToken)\n at Microsoft.Bot.Builder.Integration.ApplicationInsights.Core.TelemetryInitializerMiddleware.OnTurnAsync(ITurnContext context, NextDelegate nextTurn, CancellationToken cancellationToken)\n at Microsoft.Bot.Builder.BotFrameworkAdapter.TenantIdWorkaroundForTeamsMiddleware.OnTurnAsync(ITurnContext turnContext, NextDelegate next, CancellationToken cancellationToken)\n at Microsoft.Bot.Builder.MiddlewareSet.ReceiveActivityWithStatusAsync(ITurnContext turnContext, BotCallbackHandler callback, CancellationToken cancellationToken)\n at Microsoft.Bot.Builder.BotAdapter.RunPipelineAsync(ITurnContext turnContext, BotCallbackHandler callback, CancellationToken cancellationToken)"

Expected behavior

The entity is recognized like all other entites (and even similar entites). e.g. “Sicherheitsdienstmitarbeiterinnen” leads to the error, while “Sicherheitsdienst Mitarbeiterinnen” works perfectly fine.

[bug]

[lauren-mills]

Looks like this will be resolved by Microsoft/botbuilder-dotnet#4456.

[hhasenauer]

@mrivera-ms Tested with 4.11.1, I am still having the same issue.