Melvin and Bespoken

With Melvin, voice unlocks an array of new use cases in the genomics domain. Melvin has been an early mover in this space, and with it, we have seen the possibilities as well as become acutely aware of the limitations of these cutting edge new technologies. To overcome these challenges, Melvin used Bespoken’s software to bring state-of-the-art testing and automation practices to voice. The ultimate goal was to guarantee exceptional quality for its users' voice interaction experiences.

Unit Testing

To verify the performance and correct functionality of individual components of the code as we were developing it - much before we could actually speak to it, we leveraged the Bespoken's Unit Testing Framework. Bespoken’s Unit Tests are written in our simple YAML scripting syntax, making it easy to learn and maintain. The tests are run with a specialized version of Jest, that combines unit tests, code coverage and easy-to-use mocks in one place. In addition, Bespoken's software called Virtual Alexa is used to generate JSON requests and emulate Alexa’s behavior. This provided the best of all worlds - a full-featured general testing framework tailored to work specifically with our skill.

The unit tests were written alongside the development of the skill to ultimately cover over 90% of the code of the entire skill. Greater code coverage means greater assurance that each individual component of the skill works perfectly and ensures that future changes to the code base do not introduce any unintended side effects, significantly reducing the code-level testing effort required in the future.

End-To-End Testing

Once the skill was in a conversational state, End-to-End (E2E) Tests had to be set up. E2E testing verifies the entire system as a whole, from front-end to external services. The core components of Bespoken’s E2E testing and monitoring are Virtual Devices. A Virtual Device works like a physical device, such as an Amazon Echo, but with the important difference that it exists only as software. Virtual Devices allow interacting with Alexa just by typing.

Once the tests are written (in YAML), the statements from that file are converted into audio using Amazon Polly text-to-speech. The audio is then sent to virtual Alexa devices as inputs. The responses from these devices include an audio (the vocal response) and the metadata (to be used as display, if the device can provide a visual output as well). These responses are converted into text using speech-to-text and compared with the expected output specified in YAML Script to generate test reports.

Continuous Testing

Once the hurdle of creating automated tests was cleared, it was relatively easy to tie these unit and E2E tests into a Continuous Integration and Development tool. Having this automated process in place removed the need for human oversight into new code deployment and the need for human testers to actually speak to devices throughout the day to have confidence that it is still functioning as designed, and enabled the team to focus on the critical tasks of enhancing the skill’s capabilities.

Results

• 100% time savings on existing manual testing
• From 0 - ~90% unit-test coverage
• Hundreds of test cases run twice weekly, ensuring total system quality
• E2E testing time reduced from 1 week to 30 minutes
• Found and fixed several(10+) bugs while setting up the unit tests And last but not the least, the boost in confidence to launch the voice skill!!

Check out our case-study with Bespoken here!

To work with Bespoken, and learn more about how you can gain these benefits for your own project, just contact via email or chat.