feature request for handling callbacks

I think callback functions are also needed to use semantic-kernel better. same concept here(link)

This feature will help solving TODOs. example is here semantic-kernel/python/semantic_kernel/ai/open_ai/services/open_ai_chat_completion.py

    async def complete_chat_async(
        self, messages: List[Tuple[str, str]], request_settings: ChatRequestSettings
    ) -> str:
        ...

        # TODO: tracking on token counts/etc.

        return response.choices[0].message.content

At least i hope SKContext or ContextVariables has callback functions like on_update or on_setitem

If MS doesn’t have a plan for this feature or it takes time, I can try.

[ianchi]

Hi @alexchaomander, any comments on the proposal and PR? Having no way to track what’s going and “debug” is a big stopper for me.

[ianchi]

This is a possible structure for a handler, with hooks in the order they will be called:

class CallbackHandlerBase:

    def on_pipeline_start(self, context: SKContext):
        pass

    def on_function_start(self, context: SKContext, func: SKFunctionBase):
        pass

    #only for semantic functions
    def on_prompt_rendered(self, context: SKContext, func: SKFunctionBase, prompt: Union[str, List[Tuple[str, str]]]):
        pass

    #only if there is an error
    def on_function_error(self, context: SKContext, func: SKFunctionBase):
        pass

    def on_function_end(self, context: SKContext, func: SKFunctionBase):
        pass

    def on_pipeline_end(self, context: SKContext):
        pass

Some additional hooks should be added for the planner. The pipeline has almost no information, perhaps the call to run_async could have an optional legend parameter just to identify a specific pipeline, and pass it as an argument to the handler, to be able to have a clearer trace. The SKFunctions already have their identification in the definition.

The handlers could be set at the kernel level, and be used as defaults, or at the call level, to override for a specific invocation. All handlers will be invoked in order.

# default handlers
kernel.set_handlers([stdout_handler, thought_process_md])

# call using default handlers
result = await kernel.run_async( sf['func1'], nf['func2'], context=context)

# specific handler for call
result = await kernel.run_async( sf['func1'], nf['func2'], context=context, handlers=[thought_process_html])

If you agree, I could send a PR with an implementation of this approach.

Perhaps later, some general on_error, on_warning, on_info could also be added and use the handler in place of the log parameter, so that more consistent control of messages is possible. A default logger_handler could be available to just write in the provided logger, and so not break the current behavior