Adaptive Card does not support speech input

(Edited by @compulim to convert into a feature request)

Background

When an Adaptive Card is presented to the user, the user cannot submit the card using microphone button.

Saying “submit” to the microphone do not submit the content of the card, instead, it send a text of “submit” to the bot.

Design notes

TBC: Few different approaches to be filled in.

“What can I say?”

We should find out what the user could say on the screen (a.k.a. commands). And for those “commands”, we don’t send it to the bot as text, we do something else (e.g. submit button).

• How does it works with Direct Line Speech?
  • DLSpeech need to support “command filtering”
  • For example
    • The screen has a submit button
    • The user say, “submit”, it should not send it as text to the bot
    • The user say, “I don’t know what to do”, it should send it as text to the bot
• Should we also support “what can we say?” without sending things to the bot?
  • Should we chalkmark say-able things on the screen?
  • How about suggested actions which are far off the screen?

“You can also say …” bot logic

The bot can use LUIS and add an alternative way for submission. For example, in the case above, instead of finding it in the dropdown list, the end-user can say, “Los Angeles timezone”. And the bot understand it is one of the dropdown.

The bot might want to update the selection in the Adaptive Card to reflect what the user submitted.

Original issue

Hi All,

I am new to speak functionality in adaptive cards. We have a scenario where a drop down has to be selected and click on submit. if user just says submit, action is being submitted but the value from the drop down is going as null.

Below is the example screenshot of the adaptive card

Adaptive card JSON is :

{
    "type": "AdaptiveCard",
    "version": "1.0",
    "speak":"<s><select_timezone></s>",
    "body": [
        {
            "type": "Container",
            "items": [
                {
                    "type": "TextBlock",
                    "text": "<select_timezone>",
                    "id": "HeaderTimeZone",
                    "size": "Medium",
                    "color": "Dark",
                    "weight": "default"
                },
                {
                    "type": "Input.ChoiceSet",
                    "placeholder": "Placeholder text",
                    "id": "<DDLTimeZone>",
                    "value": "",
                    "choices": []
                   
                }
            ]
        }
    ],
    "actions": [
        {
            "type": "Action.Submit",
            "title": "<submit>",
            "Id": "SubmitTimeZone",
            "data": {
                "action": "TimeZone"
            }
        }
    ],
    "$schema": "http://adaptivecards.io/schemas/adaptive-card.json"
}

Below is the code example for binding data to adaptive card and return adaptive card

var adaptiveCardPath = File.ReadAllText(Directory.GetCurrentDirectory() + "\\PATH\\");
StringBuilder stringBuilderJson = new StringBuilder(adaptiveCardPath);
stringBuilderJson.Replace("<select_timezone>", _responseManager.GetResponse(HourGlassResponses.SelectTimeZone).Text);
stringBuilderJson.Replace("<submit>", _responseManager.GetResponse(HourGlassResponses.Submit).Text);
stringBuilderJson.Replace("<DDLTimeZone>", "DDLTimeZone");

JObject hoursObject = JObject.Parse(stringBuilderJson.ToString());
var bodyContainer = hoursObject["body"].Where(x => x["type"].Value<string>() == "Container").ToList().First()["items"];
var choiceSet = bodyContainer.Where(x => x["type"].Value<string>() == "Input.ChoiceSet");
JArray choices = choiceSet.First()["choices"] as JArray;

foreach (TimeZoneInfo time in TimeZoneInfo.GetSystemTimeZones())
{
    choices.Add(new JObject(new JProperty("title", time.DisplayName)
        , new JProperty("value", time.Id)));
}
choiceSet.First()["value"] = choices.First()["value"];

var adaptiveCardAttachment = new Attachment()
{
    ContentType = "application/vnd.microsoft.card.adaptive",
    Content = hoursObject,
};
await stepContext.Context.SendActivityAsync(MessageFactory.Attachment(adaptiveCardAttachment, inputHint: InputHints.ExpectingInput), cancellationToken);
           
we are using Microsoft custom speech service and the code is as below :

window.WebChat.renderWebChat({
directLine: window.WebChat.createDirectLine({ token: 'toren' }),
userID: id,
username: id,
locale: 'en-US',
styleOptions: styleOptions,
store: store,
selectVoice: (voices, activity) =>
global_voiceModel == "BenjaminRUS" ? voices.find(({ name }) => /BenjaminRUS/iu.test(name))
: voices.find(({ name }) => /JessaRUS/iu.test(name)),
webSpeechPonyfillFactory: createPonyfillFactory()
}, document.getElementById('webchat'));
document.querySelector('#webchat > *').focus();

function createPonyfillFactory() {
const speechServicesPonyfillFactory = window.WebChat.createCognitiveServicesSpeechServicesPonyfillFactory({
authorizationToken: user.voiceToken,
region: 'northeurope',
});

return (options) => {
const speech = speechServicesPonyfillFactory(options);

return {
  SpeechGrammarList: speech.SpeechGrammarList,
  SpeechRecognition: speech.SpeechRecognition,

  speechSynthesis: global_enableSpeech ? speech.speechSynthesis : null,
  SpeechSynthesisUtterance: global_enableSpeech ? speech.SpeechSynthesisUtterance : null
};
}
}

Thanks in Advance

(Edited by @compulim for code formatting)

[saikiran521]

@compulim - thank you for your response we will try and get back to you

[compulim]

Now I understand what you mean.

Unfortunately, currently Adaptive Cards doesn’t support speech input. Thus, if the user say “submit”, it don’t tap the submit button for them. It just send “submit” as text to the bot.

Let me convert this bug into a feature request. (And please vote on this one, to make this higher priority during our planning)