Unit testing

Adding the ability to perform unit tests easily. Here’s how I have tests setup now using @bespoken tools and ava. This isn’t the exact code behind it but an overview of it and how I’m currently using it.

SkillMock API

All the credit goes to @bespoken, I just converted what they built to be promised based so it was easier for me to work with.

const skill = new SkillMock(/* app id */)

// Starts and stops Alexa and the lambda server
skill.start(), skill.stop()

// launches the voice assistant (aka 'Alexa, ', 'Hey Google')
skill.launched()

// ends the session
skill.sessionEnded(reason)

// run intent by name and pass slots into them
skill.intended(intent, slots)

// run intents by passing in the phrases they're associated with
skill.spoken(utterance)

// set the access token for the app
skill.setAccessToken(token)

// This one isn't apart of bespoken tools. It kicks off tests ability to test easily via the response 
// that's stored when `skill.launched`, `skill.intended`, `skill.spoken`, or `skill.sessionEnded`. 
// This just returns the class that I built to help with testing
skill.test(assertion)

Testing API

// skill.test() exposes this api
// All these methods and getter functions are chainable, and they all start from skill.test()
// here's a list of assertion methods that can be used in each context
.is(str) // Assert that the value is the same as the output speech
.truthy() // Assert that the response value is truthy
.falsy() // Assert that the response value is falsy
.not(str) // Assert that the value is not the same as the output speech
.matches(regex), .match(regex) // Assert that the output speech matches regex
.notMatches(regex), .notMatch(regex) // Assert that the output speech doesn't match regex
.startsWith(str) // Assert that the output speech starts with the string
.notStartsWith(str) // Assert that the output speech doesn't start with the string
.endsWith(str) // Assert that the output speech ends with the string
.notEndsWith(str) // Assert that the output speech doesn't end with the string
.includes(str), .contains(str) // Assert that the output speech contains string
.notIncludes(str), .notContains(str) // Assert that the output speech doesn't contain string
.ended() // Assertion to ensure the session has ended
.notEnded() // Assertion to ensure the session hasn't ended

// These getter methods will change the style. So if you want to test against ssml or plain text you have the option
.ssml // changes the context style to ssml (default)
.plain // changes the context style to plain

// These getter methods change the context to test different aspects of the voice app
// all these work with the methods above
.response // changes the context to response (default)
.reprompt // changes the context to reprompt

.sessionAttributes, .attr, .attributes // changes the context to attributes
  // on top of the default assertions here are some attr specific assertions
  .attr.keys(...keys), .attr.key(...keys) // Assert that the keys exist
  .attr.notKeys(...keys), .attr.notKey(...keys) // Assert that the key doesn't exists
  .attr.is(key, expected), .attr.value(key, expected) // Assert that the key value is same as expected
  .attr.not(key, expected), .attr.notValue(key, expected) // Assert that the key value isn't the same as expected
  .attr.type(key, type) // Assert that the key's value is same as type that was passed (`[]` = 'array', `null` = 'null')
  .attr.notType(key, type) // Assert that the key's value isn't the same as the type that was passed
  .attr.truthy(key) // Assert that the key's value is truthy
  .attr.falsy(key) // Assert that the key's value is falsy

.card // changes the context to cards
  card.title // changes the context to the cards title (default)
  card.type // changes the context to the cart type
  card.image.small, card.small // changes the context to small card image
  card.image.large, card.large // changes the context to small card image
  card.text // changes the content of the card

Here’s a basic example of how it looks in a test

test('basic', async (t) => {
  const skill = new SkillMock()
  await skill.start()
  skill.setAccessToken(await mockAccessToken())

  await skill.launched() // Alexa, open [my app]
  await skill.spoken('what\'s my cashback balance')

  skill.test(t)
    .matches(/^your cashback balance is \$[0-9.]+\./) // ensure the response matches this format
    .reprompt
    .is('what else can i help you with?') // ensure the reprompt is exactly this

  await skill.stop()
})

alexa json response for basic

{
  "version": "1.0",
  "response": {
    "shouldEndSession": false,
    "outputSpeech": {
      "type": "SSML",
      "ssml": "<speak> your cashback balance is $10.20. what else can i help you with? </speak>"
    },
    "reprompt": {
      "outputSpeech": {
        "type": "SSML",
        "ssml": "<speak> what else can i help you with? </speak>"
      }
    }
  },
  "sessionAttributes": {
    "shopper_id": "112341234jlkajfasda2431asddf",
    "customer_id": "5559316",
    "distributor_id": "123412341234",
    "portal_id": 132241234,
    "language": "ENG",
    "country": "USA",
    "gender": "Male",
    "shopper_name": "John Doe",
    "STATE": "_MAIN"
  }
}

Here’s more of an advanced use case

test('advanced', async (t) => {
  const expected_prompt = 'would you like to order this product?'
  const skill = new SkillMock()
  await skill.start()
  skill.setAccessToken(await mockAccessToken())

  await skill.launched() // Alexa, open [my app]
  await skill.spoken('search for {Xbox One X}')

  skill.test(t)
    .includes('Microsoft <break time="100ms" /> Xbox One X') // ensure there's a breaktime in there after the product name. It makes it sound better
    .plain // switch to plain context
      .matches(/is \$(?:[0-9]{2,}|[1-9]+).[0-9]{2} usd/i) // ensure the price is formatted correctly on the response
      .includes(expected_prompt) // ensure the prompt is apart of the initial response
    .reprompt // switch to the reprompt context
      .is(expected_prompt) // ensure the reprompt matches prompt
    .card // switch to the card context
      .matches(/is \$(?:[0-9]{2,}|[1-9]+).[0-9]{2} usd/i) // ensure the price is formatted correctly
    .text // switch to the `card.text` context
      .notIncludes(expected_prompt) // ensure the card text doesn't include the prompt message
    .image
      .small // switch to the small image context
        .truthy() // ensure the small image exists
        .matches(/__300x300__\.jpg$/i) // ensure the small image is the 300 size
      .large // switch to the large image context
        .truthy() // ensure the large image exists
        .matches(/__600x600__\.jpg$/i) // ensure the small image is the 600 size
    .attr // switch to the attributes context
      .truthy('product') // esnure the product key exists
      .type('product', 'object') // ensure the product key is an object

  await skill.stop()
})

alexa json response for advanced

{
  "version": "1.0",
  "response": {
    "shouldEndSession": false,
    "outputSpeech": {
      "type": "SSML",
      "ssml": "<speak> Microsoft <break time=\"100ms\" /> Xbox One X - is $499.00 usd. would you like to order this product? </speak>"
    },
    "reprompt": {
      "outputSpeech": {
        "type": "SSML",
        "ssml": "<speak> would you like to order this product? </speak>"
      }
    },
    "card": {
      "type": "Standard",
      "title": "Microsoft Xbox One X is $499.00 USD.",
      "image": {
        "smallImageUrl": "https://img.shop.com/Image/240000/243400/243416/products/1588637534__300x300__.jpg",
        "largeImageUrl": "https://img.shop.com/Image/240000/243400/243416/products/1588637534__600x600__.jpg",
      },
      "text": "Games play better on Xbox One X. Experience 40% more power than any other console 6 teraflops of graphical processing power and a 4K Blu-ray player provides more immersive gaming and entertainment Play with the greatest community of gamers on..."
    }
  },
  "sessionAttributes": {
    "shopper_id": "asdasdfas31sdq14gasdasfd",
    "customer_id": "12341242994",
    "portal_id": 1234123,
    "language": "ENG",
    "country": "USA",
    "gender": "Male",
    "shopper_name": "john doe",
    "STATE": "_SEARCH_RESULTS",
    "product": {
      "id": 112312341234,
      "category_id": 12342,
      "review_count": 341,
      "pricing": "499.00",
      "is_accessory": false,
      "prod_container_id": 1123412341234
      "merchant_sku": "33414",
      "product_category_id": 12341,
      "prod_id": "1588637534",
      "category_name": "Electronic",
      "rating": 5,
      "container_text": "Microsoft Xbox One X",
      "container_text_phonetic": "Microsoft <break time=\"100ms\" /> Xbox One X",
      "discount_percentage": 0,
      "description": "Games play better on Xbox One X. Experience 40% more power than any other console 6 teraflops of graphical processing power and a 4K Blu-ray player provides more immersive gaming and entertainment Play with the greatest community of gamers on...",
      "image": {
        "small": "https://img.shop.com/Image/240000/243400/243416/products/1588637534__300x300__.jpg",
        "large": "https://img.shop.com/Image/240000/243400/243416/products/1588637534__600x600__.jpg",
      },
      "on_sale": false,
      "locale_id": 10,
      "cashback": "0.47",
      "store_name": "Walmart",
      "currency_code": "USD"
    }
  }
}

Example of how my test files are setup

Real world use case

import SkillMock, { mockAccessToken } from '../../skill-mock'
import ava from 'ava-spec'
const test = ava.group('handlers:main:cashback')

test.beforeEach(async (t) => {
  // initialize the skill (aka `bst.LambdaServer` `bst.BSTAlexa`)
  t.context.skill = new SkillMock()
  // start the skill starts the lambda and alexa servers
  await t.context.skill.start()
  // set the access token for my app. We have our own service behind the scenes that generates it for us.
  t.context.skill.setAccessToken(await mockAccessToken())
})

test.afterEach(async (t) => {
  // after each test stop the skill (aka the lambda and alexa servers)
  await t.context.skill.stop()
})


test.group('CashbackBalanceCheck', (test) => {
  const utterance = 'what is my cashback'

  test('success', async (t) => {
    const skill = t.context.skill
    await skill.launched() // launch the skill (aka `Alexa, `, `Hey Google`,)
    await skill.spoken(utterance) // pass in the utterance that's being tested 
    // since I was already using ava I just pass in their assertion lib to the test class I wrote.
    skill.test(t)
      .plain // convert the response to plain text (strip out the ssml)
      .matches(/^your cashback balance is \$[0-9.]+\./) // check to see if the response matches this regex
      .reprompt // move on to the reprompt
      .is('what else can i help you with?')  // ensure the reprompt is exactly `'what else can i help you with?'`
  })

  test('no cashback', async (t) => {
    const skill = t.context.skill
    // I had to test to what no cashback response looked like which required me use 
    // a different email address so instead of using the initial `accessToken` I set a different one
    skill.setAccessToken(await mockAccessToken('[e-mail]'))
    await skill.launched()
    await skill.spoken(utterance)
    skill.test(t)
      .plain
      .includes('you do not currently have any cashback') // the response includes this text
      .reprompt
      .is('what else can i help you with?') // the response is exactly this text
  })

  test('error', async (t) => {
    const skill = t.context.skill
    await skill.launched()
    // cause an error by setting shopper_id to be wrong
    skill.attributes.shopper_id = 'asdfasdfasdfawasdf'
    await skill.spoken(utterance)
    skill.test(t)
      .plain
      .is('something went wrong while trying to get your cashback balance. what else can i help you with?') // ensure the response is this exact text
  })
})

These don’t cover every single thing you could test but I think it’s a good starting point. I took the great work that bespoken has already done and tried to simplify it to make it a little more stream line for my use. Then I made the test interface to make my tests more readable, I modeled it after nixt which is a cli testing framework that simplifies testing cli tools. There’s other things that could definitely be added but this covered most of my use cases.

[jkelvie]

Hi @tjbenton - @jankoenig brought this to my attention. You mentioned converting our library to promises - we have pulled out our testing piece and also “promise-fied” it here: https://github.com/bespoken/virtual-alexa

We also made instantiation simpler - there is no need to start or stop anything if you have a lambda function.

It looks like you have added other pieces as well to the skill-mock though, for covering common expectation checking scenarios? Am I understanding it correctly?

[tjbenton]

let me know, I already have the js written. I would only need to adjust a couple things to make it work.

Another though I had last night was to possibly set up a test runner similar to testcafe and have specific platforms as plugins instead of browsers as plugins