1. Brad Myers 05-440/05-640: Interaction Techniques Spring, 2016 Lecture 26: Past to Future: Artificial Intelligence (AI) in Interaction Techniques 1 © 2016 - Brad Myers

2. Announcements Schedule for final presentations posted in googledoc: https://docs.google.com/spreadsheets/d/1aHnqR0KpbJTG8UAGrX67AgIDBuRohJF-fhEgIBR3h8g/edit#gid=0 https://docs.google.com/spreadsheets/d/1aHnqR0KpbJTG8UAGrX67AgIDBuRohJF-fhEgIBR3h8g/edit#gid=0 Optional meetings with me today – still slots open 2 © 2016 - Brad Myers random numberGrouptalk lengthtentative presentation date 0.025497Watch band text entry15Wed, 4/20 0.071698Better tests30Wed, 4/20 0.103625Knobs15Mon, 4/25 0.154695Watch Tilt Text entry15Mon, 4/25 0.363748Norm Cox Wiki15Mon, 4/25 0.584315Pull-to-Refresh wiki15Mon, 4/25 0.6203Virtual Reality15Mon, 4/25 0.687627Leap Motion15Wed, 4/27 0.731451Apple Watch30Wed, 4/27 0.782284Scrolling wiki15Wed, 4/27 0.928622Scroll study15Wed, 4/27

3. Intelligent User Interfaces Lots of work in Intelligent User Interfaces in general http://iui.acm.org/2016/ - yearly since 1997 http://iui.acm.org/2016/ But most are not “interaction techniques” Also, lots of work on AI to build UIs E.g., automatic selection of interaction techniques Not covered here. See slides from 05-830slides from 05-830 I selected a few interaction techniques to cover: Speech and natural language user interfaces “Data detectors” Squiggly underlining Intelligent agents (“Clippy”) © 2016 - Brad Myers 3

4. What makes a UI “Intelligent”? © 2016 - Brad Myers 4

5. What makes a UI “Intelligent”? “a user interface (UI) that involves some aspect of Artificial Intelligence (AI or Computational Intelligence). There are many modern examples of IUIs, the most famous (or infamous) being the Microsoft Office Assistant, whose most recognizable agentive representation was called "Clippy".” – Wikipediauser interfaceArtificial IntelligenceOffice AssistantClippyWikipedia Using heuristics that may be wrong Using elaborate pattern matching algorithms Recognition-based interfaces Knowledge based interfaces © 2016 - Brad Myers 5

6. Key Evaluation Issues Trust Will users trust that the system is doing the right things? High stakes vs. low stakes interactions Components: Accuracy “False negatives” – misses something it should do “False positives” – does something it should not Often can reduce one by increasing the other “Smarter” interface lowers all errors “90% accuracy” – still gets 1 out of 10 wrong! Visibility of System Status (Nielsen heuristics) Can users tell what the system is doing? User Control (Nielsen heuristics) Can users fix it when it does the wrong thing? © 2016 - Brad Myers 6

7. Speech and natural language user interfaces Speech recognition and natural language understanding has been a CS research topic since at least the 1960’s Very slow & steady progress with machine speeds and new algorithms Now “reasonably” accurate for conventional requests for people with conventional speech Speech: two phases Recognition into a transcript Problems with words sounding alike, accents, background noise, pauses, etc. Natural ways to correct are to hyper-articulate & talk slower, which often makes recognition do worse Natural language processing Problems with common sense, references (pronouns), sentence structure, etc. It turns out that dictating is difficult while thinking Especially given the need to be error free Special “sub-languages” difficult to learn Not clear what you are allowed to say Interface needs to guide the user into saying things that will work. Used to be much more strict about what can say – e.g., collect calls “yes”/”no” © 2016 - Brad Myers 7

8. Speech & NL Key advantages: Average human’s fastest output mechanism Able to “jump around” and combine tasks Can handle ambiguity and partial descriptions Versus direct manipulation Example: “Schedule a meeting the day before CHI with everyone in my group.” Key disadvantages Inaccuracies, misrecognitions, unclear scope Difficulties of corrections when wrong © 2016 - Brad Myers 8

9. Today’s speech systems Apple Siri, Google Now Microsoft’s “Cortana” -- refref Amazon Echo & equivalent Always listening As opposed to “push to talk” like cars Video ad Recognition by the “cloud” Single statement phrases and questions Doesn’t remember a dialog “Direct answers” also available in Google © 2016 - Brad Myers 9

10. Can we get some of those advantages without speech? PhD research of Toby Li Ineffective use of smartphones Data is all “siloed” in apps Multi-app tasks Inefficient or repetitive data or command entry © 2016 - Brad Myers 10

11. “Data Detectors” Pattern matcher that looks for specific kinds of data in plain text Enables various operations on that text E.g., recognizing phone numbers, people names, URLs, email and physical addresses, etc. Nardi, B.A., Miller, J.R., and Wright, D.J. “Collaborative, programmable intelligent agents.” Comm. ACM 41, 3 (1998), pp. 96–104. “Apple Data Detectors” US 5,946,647 – “System and method for performing an action on a structure in computer-generated data” by Thomas Bonura, James R. Miller, Bonnie Nardi, David Wright, Filed: Feb 1, 1996, https://www.google.com/patents/US5946647 https://www.google.com/patents/US5946647 In the Apple v. Samsung case © 2016 - Brad Myers 11

12. Research related to Data Detectors Grammex Lieberman, H., Nardi, B.A., and Wright, D. Grammex: defining grammars by example. Demo at CHI'98, ACM (1998), pp. 11–12. http://web.media.mit.edu/~lieber/Lieberary/Grammex/Grammex-Intro.html http://web.media.mit.edu/~lieber/Lieberary/Grammex/Grammex-Intro.html Define the pattern by giving a bunch of examples Listpad – use data detectors to recognize structure in plain text lists Kerry S. Chang, Brad A. Myers, Gene M. Cahill, Soumya Simanta, Edwin Morris and Grace Lewis. "Improving Structured Data Entry on Mobile Devices", ACM Symposium on User Interface Software and Technology, UIST'13, October 8-11, 2013, St. Andrews, UK. pp. 75-84. acm dl or local pdf and video (5:00)acm dllocal pdfvideo Combine with web services to make data entry easier © 2016 - Brad Myers 12

13. Squiggly (Wavy) underlining In Word: for misspellings (red), grammar problems (green), and formatting problems (Blue) – referencereference Introduced in Word 95 for Windows – citecite Originally, grammar checker was quite bad, but significantly improved over time Too many “false positives” AI researchers at Microsoft Research helped with better language models Now used for errors in code as well as regular documents © 2016 - Brad Myers 13

14. Intelligent Agents © 2016 - Brad Myers 14 A common aspiration of AI is a personified agent E.g., “Knowledge Navigator” video from Apple, 1987 video Microsoft’s “Office Assistant”, known as “Clippy” – video (0:38)video Office 1997 to 2003 “Smithsonian Magazine called Clippy “’one of the worst software design blunders in the annals of computing’". – citecite Too often useless and wrong (false positives) Animates even when you are not supposed to use it. Whole thesis on “Why People Hate the Paperclip: Labels, Appearance, Behavior and Social Responses to User Interface Agents” – pdfpdf Clifford Nass’s work on personification

15. Automatic Design of UIs Specify some property of design and system picks widgets Simple example: html Long history of more elaborate rules © 2016 - Brad Myers 15

16. Early Example: Mickey D. R. Olsen, Jr.. 1989. A programming language basis for user interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '89), K. Bice and C. Lewis (Eds.). ACM, New York, NY, USA, 171-176. http://dx.doi.org/10.1145/67449.67485 http://dx.doi.org/10.1145/67449.67485 Programmer specifies UI by putting special comments in a Pascal file. Uses the Apple Macintosh guidelines Pre-processor to parse the Pascal code and generate the Macintosh resources. Maps Procedures into Menu items. If parameter is one of a standard set, pops up appropriate dialog box or waits for input File to be read, file to be written New point, line or rectangle 16 © 2016 - Brad Myers

17. Personal Universal Controller Jeff Nichols’ PhD work (2006)PhD work Problem: Appliance interfaces are too complex and too idiosyncratic. Solution: Separate the interface from the appliance and use a device with a richer interface to control the appliance: PDA, mobile phone, etc. Goal: Generate high-quality UIs 17 © 2016 - Brad Myers

18. Control Feedback Approach Specifications Appliances Mobile Devices Use mobile devices to control all appliances in the environment Key Features Two-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation 18 © 2016 - Brad Myers

19. Properties of PUC Language State variables & commands Each can have multiple labels Useful when not enough room Typed variables Base types: Boolean, string, enumerated, integers, fixed-point, floating-point, etc. Optional labels for values Hierarchical Structure Groups 19 © 2016 - Brad Myers

20. Generating Consistent UIs Personally consistent Reduce learning time Add unique functions 20 © 2016 - Brad Myers

21. Generating Combined UIs For multiple appliances, such as home theaters Specify content flow Combined controls 21 © 2016 - Brad Myers

22. Summative Study Compared PUC to manufacturer’s interfaces for HP and Canon printer/fax/copiers PUC twice as fast, 1/3 the errors Consistent: another factor of 2 faster 22 © 2016 - Brad Myers

23. Recommender Systems Collaborative filtering – based on other people’s choices, as well as yours Dates back to early research at Xerox PARC (1992) David Goldberg, David Nichols, Brian M. Oki, Douglas Terry, Using collaborative filtering to weave an information tapestry, Communications of the ACM, v.35 n.12, p.61-70, Dec. 1992. http://dl.acm.org/citation.cfm?id=245121http://dl.acm.org/citation.cfm?id=245121 Also content-based or hybrid © 2016 - Brad Myers 23

24. Other examples Personalized autocomplete like Google search bar Natural language like google search bar, etc. Programming-by-example in general User models, like for intelligent tutors Robotics – Roomba (not really an interaction technique) … © 2016 - Brad Myers 24