1. Kyungmin Lee *, Jason Flinn *, T.J. Giuli +, Brian Noble *, and Christopher Peplin + University of Michigan * Ford Motor Company + AMC: Verifying User Interface Properties for Vehicular Applications

2. Where do we use mobile apps? 2 Kyungmin Lee WalkingDrivingEverywhere we go (Not-too-distant future)

3. Where do we use mobile apps? 3 Kyungmin Lee WalkingDrivingEverywhere we go (Not-too-distant future)

4. Mobile apps in a vehicle 4 Pros: - High usability standard - Good form factor Cons: - Cost of deployment - Limited applications Pros: - Diverse applications - Easy deployment Cons: - Distracting apps - Poor form factor Kyungmin Lee

5. Mobile apps in a vehicle 5 Pros: - High usability - Good form factor Pros: - Diverse apps - Easy deployment Kyungmin Lee How can we get the best of both worlds?

6. Our approach Create a tool (AMC) that – Automatically explores the application – No source code & app-specific knowledge – Pinpoints UI guideline violations 6 Kyungmin Lee

7. AMC’s benefits Developers – Test own apps – Early design feedback 7 User interface experts – Fewer properties to check – Focus on hard properties Kyungmin Lee

8. AMC’s benefits Developers – Test own apps – Early design feedback 8 User interface experts – Fewer properties to check – Focus on hard properties Kyungmin Lee 20X reduction in testing False negative: 2% False positive: 0% 20X reduction in testing False negative: 2% False positive: 0%

9. Outline Motivation How AMC explores the app How AMC finds UI violations How closely do apps follow UI guidelines Conclusion 9 Kyungmin Lee

10. How to use AMC 10 Run AMCGet violation report Violation report Button size: 1) Button A at state 1 2) … Contrast ratio: 1)… 2)… Violation report Button size: 1) Button A at state 1 2) … Contrast ratio: 1)… 2)… Kyungmin Lee - State exploration - Find violations - State exploration - Find violations Application Monkey Hierarchy Viewer Modified Applicatio nFramewo rk Modified Applicatio nFramewo rk App to test

11. How does AMC explore? 11 State #0 Kyungmin Lee

12. How does AMC explore? 12 State #0 Current Location State #1 Kyungmin Lee

13. How does AMC explore? 13 Kyungmin Lee

14. How do we define a state Different screens -> Different states 14 Kyungmin Lee vs.

15. Fine-tuning state definition Naïve approach: every unique screen -> state Solution: Structure hash – Same structure, different content -> same state – Avoids state explosion 15 vs. Kyungmin Lee

16. Fine-tuning state definition Structure hash is not enough… Solution: Consider only 0,1,or many items – If there are 2+ items, ignore the rest – Unless they transition to different states 16 vs. Kyungmin Lee

17. Outline Motivation How AMC explores the app How AMC finds UI violations How closely do apps follow UI guidelines Conclusion 17 Kyungmin Lee

18. Finding UI violations Checks for 7 UI properties – User actions per task, Text contrast ratio, Word count, Button size, Button distance, Animation, Scrolling – Defined by transportation research groups – Report violation, no violation, or defer to experts 18 Kyungmin Lee

19. Finding UI violations Properties of individual state 19 Kyungmin Lee Button size Button closeness Text contrast ratio Word count Animation Scrolling

20. Finding UI violations Properties of individual state 20 Kyungmin Lee Button size ✔ Button closeness ✔ Text contrast ratio Word count Animation Scrolling

21. Finding UI violations Properties of individual state 21 Kyungmin Lee Button size ✔ Button closeness ✔ Text contrast ratio ✖ Word count Animation Scrolling

22. Finding UI violations Properties of individual state 22 Kyungmin Lee Total of 12 words Button size ✔ Button closeness ✔ Text contrast ratio ✖ Word count ✔ Animation Scrolling

23. Finding UI violations Properties of individual state 23 Kyungmin Lee Button size ✔ Button closeness ✔ Text contrast ratio ✖ Word count ✔ Animation ✔ Scrolling ✔

24. Checking user actions per task 24 Kyungmin Lee

25. Few complications Buttons in custom view – Empirically discover buttons – If fails, defer to manual verification Determining good vs. bad animation 25 Kyungmin Lee vs.

26. Outline Motivation How AMC explores the app How AMC finds UI violations How closely do apps follow UI guidelines Conclusion 26 Kyungmin Lee

27. Evaluation Environmental setup 27 Kyungmin Lee

28. Comparison with expert’s evaluation 28 ApplicationWord countButton sizeButton distance AnimationScrolling Google Finance MyDays eBay Twitter GasBuddy Google Nav. TomTom Nav. Waze iRadar Beat the Traffic Best Parking MPG Kyungmin Lee Tested 12 out of 14 apps & 5 out of 7 properties Tested 12 out of 14 apps & 5 out of 7 properties

29. Comparison with expert’s evaluation 29 ApplicationWord countButton sizeButton distance AnimationScrolling Google Finance MyDays eBay Twitter GasBuddy Google Nav. TomTom Nav. Waze iRadar Beat the Traffic Best Parking MPG Kyungmin Lee : Violation : No violation : Maybe 1 st column: AMC’s report 2 nd column: Expert’s report

30. Comparison with expert’s evaluation 30 ApplicationWord countButton sizeButton distance AnimationScrolling Google Finance MyDays eBay Twitter GasBuddy Google Nav. TomTom Nav. Waze iRadar Beat the Traffic Best Parking MPG Kyungmin Lee : Violation : No violation : Maybe 1 st column: AMC’s report 2 nd column: Expert’s report

31. Comparison with expert’s evaluation 31 ApplicationWord countButton sizeButton distance AnimationScrolling Google Finance MyDays eBay Twitter GasBuddy Google Nav. TomTom Nav. Waze iRadar Beat the Traffic Best Parking MPG Kyungmin Lee : Violation : No violation : Maybe 1 st column: AMC’s report 2 nd column: Expert’s report

32. Comparison with expert’s evaluation 32 ApplicationWord countButton sizeButton distance AnimationScrolling Google Finance MyDays eBay Twitter GasBuddy Google Nav. TomTom Nav. Waze iRadar Beat the Traffic Best Parking MPG Kyungmin Lee : Violation : No violation : Maybe 1 st column: AMC’s report 2 nd column: Expert’s report Agreed 50 / 60 cases ≈ 83% Agreed 50 / 60 cases ≈ 83%

33. Comparison with expert’s evaluation 33 ApplicationWord countButton sizeButton distance AnimationScrolling Google Finance MyDays eBay Twitter GasBuddy Google Nav. TomTom Nav. Waze iRadar Beat the Traffic Best Parking MPG Kyungmin Lee : Violation : No violation : Maybe 1 st column: AMC’s report 2 nd column: Expert’s report One case of false negative <≈ 2% One case of false negative <≈ 2%

34. Comparison with expert’s evaluation 34 ApplicationWord countButton sizeButton distance AnimationScrolling Google Finance MyDays eBay Twitter GasBuddy Google Nav. TomTom Nav. Waze iRadar Beat the Traffic Best Parking MPG Kyungmin Lee : Violation : No violation : Maybe 1 st column: AMC’s report 2 nd column: Expert’s report

35. Comparison with expert’s evaluation 35 ApplicationWord countButton sizeButton distance AnimationScrolling Google Finance MyDays eBay Twitter GasBuddy Google Nav. TomTom Nav. Waze iRadar Beat the Traffic Best Parking MPG Kyungmin Lee : Violation : No violation : Maybe 1 st column: AMC’s report 2 nd column: Expert’s report Animation: 4 out of 381 states Button size & distance: 13 states Reduced by 95%! Animation: 4 out of 381 states Button size & distance: 13 states Reduced by 95%!

36. Assessment of Android apps 36 ApplicationUser actions per task Text contrast ratio Word countButton sizeButton distance Animation# Item to scroll Gmail Google Finance MyDays eBay Twitter GasBuddy Google Maps Google Nav. TomTom Nav. Waze iRadar Beat the Traffic Best Parking MPG Kyungmin Lee

37. Assessment of Android apps 37 ApplicationUser actions per task Text contrast ratio Word countButton sizeButton distance Animation# Item to scroll Gmail 17220003 Google Finance 0131403 MyDays 20420201 eBay 0110002 Twitter 2428701409 GasBuddy 1214101304 Google Maps 8730701 Google Nav. 0002100 TomTom Nav. 26061006 Waze 721501504 iRadar 0215400 Beat the Traffic 5700100 Best Parking 0000101 MPG 0000000 Kyungmin Lee

38. Assessment of Android apps 38 ApplicationUser actions per task Text contrast ratio Word countButton sizeButton distance Animation# Item to scroll Gmail 17220003 Google Finance 0131403 MyDays 20420201 eBay 0110002 Twitter 2428701409 GasBuddy 1214101304 Google Maps 8730701 Google Nav. 0002100 TomTom Nav. 26061006 Waze 721501504 iRadar 0215400 Beat the Traffic 5700100 Best Parking 0000101 MPG 0000000 Kyungmin Lee

39. Assessment of Android apps 39 ApplicationUser actions per task Text contrast ratio Word countButton sizeButton distance Animation# Item to scroll Gmail 17220003 Google Finance 0131403 MyDays 20420201 eBay 0110002 Twitter 2428701409 GasBuddy 1214101304 Google Maps 8730701 Google Nav. 0002100 TomTom Nav. 26061006 Waze 721501504 iRadar 0215400 Beat the Traffic 5700100 Best Parking 0000101 MPG 0000000 Kyungmin Lee 0.75 violations / state

40. Assessment of Android apps 40 ApplicationUser actions per task Text contrast ratio Word countButton sizeButton distance Animation# Item to scroll Gmail 17220003 Google Finance 0131403 MyDays 20420201 eBay 0110002 Twitter 2428701409 GasBuddy 1214101304 Google Maps 8730701 Google Nav. 0002100 TomTom Nav. 26061006 Waze 721501504 iRadar 0215400 Beat the Traffic 5700100 Best Parking 0000101 MPG 0000000 Kyungmin Lee

41. Assessment of Android apps 41 ApplicationUser actions per task Text contrast ratio Word countButton sizeButton distance Animation# Item to scroll Gmail 17220003 Google Finance 0131403 MyDays 20420201 eBay 0110002 Twitter 2428701409 GasBuddy 1214101304 Google Maps 8730701 Google Nav. 0002100 TomTom Nav. 26061006 Waze 721501504 iRadar 0215400 Beat the Traffic 5700100 Best Parking 0000101 MPG 0000000 Kyungmin Lee 0.69 violations / state

42. Assessment of Android apps 42 ApplicationUser actions per task Text contrast ratio Word countButton sizeButton distance Animation# Item to scroll Gmail 17220003 Google Finance 0131403 MyDays 20420201 eBay 0110002 Twitter 2428701409 GasBuddy 1214101304 Google Maps 8730701 Google Nav. 0002100 TomTom Nav. 26061006 Waze 721501504 iRadar 0215400 Beat the Traffic 5700100 Best Parking 0000101 MPG 0000000 Kyungmin Lee Only MPG app had no violations

43. Conclusion AMC is a tool that – Explores the application automatically – Pinpoints UI guideline violations Many vehicular apps with UI violations – Shows the need for a tool like AMC Can be extended to other UI settings 43 Kyungmin Lee

44. 44 Questions? Kyungmin Lee

45. 45 Kyungmin Lee

46. Backup slides 46 Kyungmin Lee

47. State coverage 47 Avg. coverage: 80% Kyungmin Lee

48. States found & runtime 48 Application# of States found Runtime (h:mm) Application# of States found Runtime (h:mm) Gmail270:56Google Nav.150:37 Google Finance 180:55TomTom Nav.364:05 MyDays261:55Waze816:09 eBay240:50iRadar231:05 Twitter6510:25Beat the Traffic 180:19 GasBuddy594:28Best Parking110:10 Google Maps311:59MPG51:22 Avg. runtime: 2hr 31minMedian runtime: ~74 minutes Kyungmin Lee

49. Non-deterministic transitions Caused by – Configuration option – Previous paths – Mistakes in state classification Solution – Avoid using non-deterministic transitions! – Use past exploration information Combine previous N transitions into one Attempt to make it deterministic 49 Kyungmin Lee

50. Non-deterministic transitions Caused by unaware hidden property 50 BACK Kyungmin Lee

51. Non-deterministic transitions Caused by our heuristics 51 ≈≠ Kyungmin Lee

52. Related work Model checking to find software bugs – KLEE, EXE, eXplode, CMC GUI Model checking with pre-specified model – Dwyer et al., Paiva et al., Takala et al. GUI checking with no model – GUI Ripping, Amalfitano et al., AutoBlackTest Using cognitive model to predict distraction – Salvucci’s work 52 Kyungmin Lee

53. Limitations Only two ways to explore the app – GUI buttons & Android actions (e.g., back button) – No voice input, multi-touch, gesture, etc No checkpoint & restore – Inability to completely revert back Relying on app’s dynamic content – Different time -> different violation result (e.g., Random pop-ups) 53 Kyungmin Lee

54. Handling text inputs Need to input password, address, etc – User pre-specifies the text – User pre-specifies the associated text field’s id AMC automatically inputs text 54 Kyungmin Lee

55. How we handle an ad No different than a button – Look for button size, button distance violations – Look for animation No violation found in our evaluation 55 Kyungmin Lee

56. Industry guideline (TRL, 02) Qualitative properties – “Color scheme must not be confusing” – “System should provide precise error message” Quantitative properties – “Text contrast ratio should be at least 3:1” – “Buttons should be at least 15mm apart” – Can be verified without human efforts 56 Kyungmin Lee