Allow ISKFunction to return IAsyncEnumerable

Is there any way currently to allow ISKFunction to return an IAsyncEnumerable? If not is there any plan to?

[MithrilMan]

Since you don’t have this yet in Semantic Kernel, let me give my view on this:

I’m working on my own implementation of generative LLM integration here: https://github.com/MithrilMan/AIdentities since a couple of months (I didn’t knew about semantic kernel and when I found it it was already late and anyway wasn’t adhering 100% to my vision) and I implemented what @wayne-o is asking in my design: I’ve a concept of Skills and when a skill is executed, this is the signature of the call IAsyncEnumerable<Thought> ExecuteAsync(Prompt prompt, SkillExecutionContext context, CancellationToken cancellationToken); The core of my skill/brain thing is here (readme isn’t updated) https://github.com/MithrilMan/AIdentities/tree/25e60f875979a0d595561fb7d0f1f5e82e5ab7a1/src/AIdentities.Shared/Features/CognitiveEngine

It works, but I’m sick of a problem that arises around the use of IAsyncEnumerable: when the API fails in the middle of generating a streamed content, it fails badly because it fails when you start to consume the iterator and .Net doesn’t have a nice way to handle such cases, so if you want to implement a retry pattern for example by using polly (and we’ve seen how this is important with OpenAI that lately had big problems serving properly their API), you can’t just wrap the call to the stream method because the exception of course arise when you materialize the enumerator. Unluckily .Net doesn’t allow to call a yield return within a try/catch block, so you have to create all sort of workaround to have a proper retry method.

Also when you stream chunks of text, what would you do if something goes wrong in the API ? In a simple chat application you can just ask the user to send again the message, but if it’s part of a skill (or skfunction as you call it) It’s very likely that You’d need to inform the consumer that you have to start over and that it should forget the partial generated text (= higher complexity)

I’ve yet not figured out how to properly and affectively handle such scenarios, in my case when it fails, fails badly, but this makes me wonder if it’s worth to use streaming calls in skills at all.

Of course it’s cool to see that my Skill is producing a text as a stream and the user can start reading as soon as it’s generating, also what I called AIdentity can generate different kind of “thoughts” that can be streemed too (think of it like a kind of log in some scanrio) but at the same time if it breaks in the middle you’ll have big troubles to rollback what you eventually did with the partial generated text, without mentioning minor problems it can give if you try to parse that text on a markdown viewer for example (that’s why recently in my chat section I’m building the message on a normal div and then swith to markdown once completed), and I’m starting to think that maybe all this complexity isn’t worth the effort.

I’m a bit torn as to whether or not implementing stream at the function level is something I’d want to deal with. Maybe just being able to signal to consumer events like “StartingTextGeneration” / “EndingTextGeneration” could be enough

[awharrison-28]

@RogerBarreto has some ideas in the works to make the requested behavior possible. TBD - when we get those changes in, let’s make sure to tag this issue.