Error synthesizing speech when including an QR-code image in the QnA maker.

Hello!

Our team was working on the project that would use the Web Bot implemented for a React application with the speech-to-text and text-to-speech functionality for the Bot connected to a QnA Maker.

I was successful in connecting the features for speech, however noticed a strange bug. When the response that we send contains a QR code in it, the speech synthesis seems to fail entirely. The qr code is embedded as an image in the QnA maker response, not sure if this would be an expected functionality. You can check it out through the following.

Navigate to:

https://wow-kiosk.netlify.app/kiosk

In the bot field you can test the speech functionality by just saying “Hello” and it should synthesize speech correctly.

However, when you ask “Tell me about Week of Welcome”, the following error pops up in the console.

I would gladly appreciate any feedback about this issue. What could be a possible go around on the issue? Should the library be able to handle embedded images?

Best,

Oleg T.

Please view our Technical Support Guide before filing a new issue.

Screenshots

Version

To determine what version of Web Chat you are running, open your browser’s development tools, and paste the following line of code into the console.

[].map.call(document.head.querySelectorAll('meta[name^="botframework-"]'), function (meta) { return meta.outerHTML; }).join('\n')

If you are using Web Chat outside of a browser, please specify your hosting environment. For example, React Native on iOS, Cordova on Android, SharePoint, PowerApps, etc.

Describe the bug

Steps to reproduce

1. Go to ‘…’
2. Click on ‘…’
3. Scroll down to ‘…’
4. See error

Expected behavior

Additional context

[Bug]

[tielushko]

@Zerryth @compulim Thank you guys for helping me figure it out. I can imagine now how that link for the image would fail to be synthesized by the speech service. I will try and find a workaround with the bot logic to prevent the link elements from being sent to the speech service for processing.

Thank you so much! Have a wonderful rest of your week!

[compulim]

@Zerryth for QnA maker, do you know if the speak field can be customized when the underlying bot is responding?

Specifically, I mean, when the bot reply (schema here), can the QnA maker reply a different value for the speak field, while keeping the text field containing the QR code?

Currently, the bot response is too long/complex for Cognitive Services to synthesis it out, copied the Activity object below:

  {
    "type": "message",
    "id": "3cg...|0000006",
    "timestamp": "2021-04-09T17:18:15.0723723Z",
    "channelId": "directline",
    "from": {
      "id": "...",
      "name": "..."
    },
    "conversation": {
      "id": "3cg..."
    },
    "locale": "en-US",
+   "text": "**Week of Welcome (WOW)** is week-long USF celebration to welcome new and returning students each fall and spring! WOW features a series of events and programs for new and returning Bulls to make lasting connections and have a successful start to your semester. New Student Connections (NSC) coordinates Week of Welcome (WOW) on the Tampa campus. SCAN THE QR Code for details!\n\n![WOW-Link](https://api.qrserver.com/v1/create-qr-code/?size=150x150&data=https://www.usf.edu/student-affairs/new-student/students/wow-fifty-days.aspx)",
    "inputHint": "acceptingInput",
    "suggestedActions": {
      "actions": []
    },
    "replyToId": "3cg...|0000005"
  }

Without the speak field set, the text field (highlighted) will be used for synthesis, which can be too long/complex. That means, we will send a SSML for the following to Cognitive Services, which is probably too long/complex for it to synthesize it.

<speak version="1.0" xml:lang="en-US">
  <voice xml:lang="en-US" name="Microsoft Server Speech Text to Speech Voice (en-US, AriaNeural)">
    <prosody pitch="+0%" rate="+0%" volume="+0%">
      **Week of Welcome (WOW)** is week-long USF celebration to welcome new and returning students each fall and spring! WOW features a series of events and programs for new and returning Bulls to make lasting connections and have a successful start to your semester. New Student Connections (NSC) coordinates Week of Welcome (WOW) on the Tampa campus. SCAN THE QR Code for details!

![WOW-Link](https://api.qrserver.com/v1/create-qr-code/?size=150x150&data=https://www.usf.edu/student-affairs/new-student/students/wow-fifty-days.aspx)
    </prosody>
  </voice>
</speak>

Thus, Cognitive Services returned 400.