Holistic Usability Measure VUI Design: Philosophy and Practice

2 of 18 VUI Design: Science vs. Art Our session was described as “... dives deep into some of the science-versus-art issues in dialog design.”

3 of 18 Art & Science in VUI Design VUI Design is both an Art and Science

4 of 18 Art & Science in VUI Design Art in VUI Design Creative uses of text/speech More than 1 way to say something Creative uses of audio “Color” in the interface Voice talent Mood of the interface

5 of 18 Art & Science in VUI Design Science in VUI Design Usability Measurable metrics Human factors Short term memory Software engineering Reuse Development

6 of 18 Art & Science in VUI Design What really matters?

7 of 18 Usability! “If you could build a system that resulted in world peace, but no one could use it... it would be useless. Usability matters.” – Gilbert However, “there has been no generally agreed-upon method of measuring usability, which makes it difficult to compare among different findings.” - Polkosky

Holistic Usability Measure

9 of 18 VUI Goals What is the goal of your VUI? Who decides the primary goal of the VUI? Designer Client Manager

10 of 18 VUI Goals Did you build the right VUI? Who decides? How do you know if your VUI is better than another that does the same tasks? You must test it!

11 of 18 Background During requirement analysis, before design Information design; what is to be built What is the VUI suppose to accomplish? Who are the users? Etc. These are typical, but most people miss Usability metrics These are typically an afterthought As a result, usability gets cut

12 of 18 Background Quantitative Metrics Speech recognition accuracy Task completion time Subjective Metrics User friendliness Ease of use

13 of 18 Background Which metrics are most important? It depends on your VUI Decide how you will test your VUI before you design it Given several different metrics How do you measure the overall effectiveness of the VUI?

14 of 18 Holistic Usability Measure HUM Assign each metric a weight (1 – 99%) based on its importance Design and implement the VUI Evaluate the VUI Convert all metrics to a similar numerical scale

15 of 18 Holistic Usability Measure HUM = W1*Metric1 + W2*Metric Wn*Metric n where W1 + W Wn = 1 and 0 ≤ W1, W2,..., Wn ≤ 1 HUM yields a percentage measure rating the VUI’s usability.

16 of 18 Conclusions VUI Design is both an Art & Science However, what really matters is usability The goodness of your VUI can only be determined through usability testing Otherwise, you are guessing Testing is essential!

17 of 18 Conclusions HUM is a tool for testing It reports usability with respect to the VUI goals Multiple ways to design same VUI. Which one is the best? HUM is ideal for comparing more than one implementation of the same VUI HUM is easy to use and it makes since Recall, “there has been no generally agreed-upon method of measuring usability, which makes it difficult to compare among different findings.” – Polkosky (SpeechTECH Magazine November/December 2005)

Thank You