`Planner` selected a wrong skill when there are two similar skills within a `skills` folder

Describe the bug When I run Example12_Planning.cs, the planner failed to complete. Below is the console log:

======== Planning - Create and Execute Poetry Plan ========
Original plan:
<goal>
Write a poem about John Doe, then translate it into Italian.
</goal>
<plan>
  <function.WriterSkill.ShortPoem input="John Doe" />
  <function.WriterSkill.TranslateV2 input="$SHORT_POEM" language="Italian" setContextVariable="TRANSLATED_POEM" />
</plan>
Step 1 - Execution results:
<goal>
Write a poem about John Doe, then translate it into Italian.
</goal><plan>
  <function.WriterSkill.TranslateV2 input="$SHORT_POEM" language="Italian" setContextVariable="TRANSLATED_POEM" />
</plan>
Step 2 - Execution results:
<goal>
Write a poem about John Doe, then translate it into Italian.
</goal><plan>
</plan>
Step 2 - COMPLETE!
Error: InvalidRequest: The request is not valid, HTTP status: BadRequest
Execution complete in 6437 ms!
fail: object[0]
      Function call fail during pipeline step 0: WriterSkill.TranslateV2. Error: InvalidRequest: The request is not valid, HTTP status: BadRequest

It appears to me that the prompt here is to translate the generated poem to Italian but because the planner chose WriterSkill.TraslateV2 (which is the skill to translate from any language to English), it failed.

Expected behavior The planner should select WriterSkill.Translate skill instead of WriterSkill.TranslateV2 and executed successfully as follows

======== Planning - Create and Execute Poetry Plan ========
Original plan:
<goal>
Write a poem about John Doe, then translate it into Italian.
</goal>
<plan>
  <function.WriterSkill.ShortPoem input="John Doe is a kind and generous man who loves to help others and make them smile."/>
  <function.WriterSkill.Translate language="Italian"/>
</plan>
Step 1 - Execution results:
<goal>
Write a poem about John Doe, then translate it into Italian.
</goal><plan>
  <function.WriterSkill.Translate language="Italian" />
</plan>
Step 2 - Execution results:
<goal>
Write a poem about John Doe, then translate it into Italian.
</goal><plan>
</plan>
Step 2 - COMPLETE!
John Doe è un uomo di grande cuore e grazia
Ha sempre un sorriso sul volto
Aiuta i poveri, i malati e i vecchi
Ma a volte la sua bontà lo mette nei guai
Come quando ha dato il suo cappotto a un tremante

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu
• IDE: VSCode

[mohamedcherifmo]

I think that’s a different error since i’m getting this error right now for services that work working earlier today… Something must be down

[lemillermicrosoft]

Yeah, it looks like it’s hallucinating the input for the TranslateV2 skill based on the examples given, a known issue. We’re looking to update things to make it so users can supply their own examples to have it be better representative of the problem space and to remove the default examples which tend to make the model follow a pattern, particularly with GPT3 and less capable models. I’d advise trying with GPT4 if possible as that was the true target for Planner capabilities. Will keep this issue open as we iterate the next few weeks though.

There are debug logs right now available about function selection:

context.Log.LogDebug(“Found relevant function. Relevance Score: {0}, Function: {1}”, memoryEntry.Relevance, function.ToFullyQualifiedName());