COGs: Towards Universal Enduring Personal Cognition Amplifiers Raj Reddy Carnegie Mellon University Sep 18, 2017 Keynote Speech at IBM Cognitive Colloquium

Desired Attributes of Intelligent Assistants Learn From Experience Exhibit Goal-directed Behavior Exploit Vast Amounts Of Knowledge Tolerate Errorful, Unexpected and Possibly Unknown Input Use Symbols and Abstractions Communicate Using Natural Language Respond in Human Reaction Time (in Milli Seconds?)

Desired Attributes of Intelligent Assistants(Cont) Get the right information e.g. search engines To the right people e.g. categorizing, routing At the right time e.g. just-in-time (task modeling, planning) In the right language e.g. machine translation In the right medium e.g. using speech for language divide and literacy divide With the right level of detail (granularity) e.g. summarization Carbonell, Language Technology Institute Manifesto, 1994

Emergence of Alternative Models Intelligent Assistants IBM Watson Deep Question Answering in Multiple Domains Apple Siri Voice based Shallow QA Microsoft Cortana Evolution of Chatbots Google Home Amazon Alexa

Emergence of Alternative Models Intelligent Assistants IBM Watson Question Answering in Multiple Domains Apple Siri Voice based Shallow QA Microsoft Cortana Evolution of Charbots Google Home Amazon Alexa

COGs: Cognition Amplifiers A Family of Enduring Personal Cognitive Assistants Link to Video: www.rr.cs.cmu.edu/ff.wmv

Kate: Good Morning! Kate here. COG must See, Hear and Interact: A Society of COGs with Specialized Functions Kate: Good Morning! Kate here. MP: Good Morning, Kate. It’s April 1, 1997 and the time is 6:30AM. Kate News Selection reports new trade agreements between Japan and France affects Paris Spring Collection plus 24 other items of interest. Care to browse? Kate: Umm. Not now. Mail only, please. FP: Mail serve here. 43 items including 26 unopened for three days, 12 unopened for two days and (interruption: Kate: cancel list) Ummm. Alright! Alright, I’ve been busy. FP: Canceling. Were there further instructions? Kate: Yes, uh, did my Snap Catalog arrive? FP: I understood “catalog”. Snap. Checking.

COG must Be Self Aware: including the hardware, software, and networking environment Kate: Yes, uh, did my Snap catalog arrive? FP: I understood "catalog". "Snap" -- checking. Kate: Alright! MP: Care to view large screen? MP: Did that mean "yes"? Kate: Yes. Switch to large screen.

COG must Be Aware of Emotions Kate: Yes, uh, did my Snap catalog arrive? FP: I understood "catalog". "Snap" -- checking. Kate: Alright! MP: Enthusiasm detected. Care to ….

COG must Know what it does not know Kate: Just do it. I'll fight that one later. MP: Searching. Did not understand last command. Kate: Do it. Do it. Do it. MP: Understood.

COG must Learn from Experience MP: Incoming Vid-Phone call. George Smith from Snap Headquarters, priority one. Do you wish to preview Mr. Smith? Kate: Darn! MP: You've used that before. Do you care to define it?

COG must Remember and Prompt at the Right Time Kate: Um...Picture Perfect, uh, Fashion Source and Alter Image. MP: You asked to be reminded. Search on Alter Image databank is very expensive and likely to exceed your monthly online budget. Cost is $485 per thousand-image scan.

MP: Alternate suggestions? COG must Provide the Right Information: in Right Language at the Right Time in the Right level of Granularity MP: James Dean image not available. Copyright Paramount Studios. Last usage granted at $15,000 to Levi Strauss for in-house promotion, one time only. Kate: Darn! MP: Alternate suggestions? Kate: Uh, yeah. Let's see...uh, image search, uh, circa 1959, uh, blue jeans, male, sex appeal. MP: Searching Nexus.. Access other databanks?

COG must Offer Right Advice in the Right Context MP: As originator of the conference call, do you wish to choose resolution on live video? Kate: Whatever. MP: Default settings confirmed. You asked to be reminded, full video from Paris is expensive at $3.55 per minute. A call of less than a minute will exceed your budget of... Kate: Got it. Um, top resolution on George, face only on Genevieve. MP: Understood. Conference call open.

COG must also possess several Implied Capabilities A Society of COG Agents with Specialized Functions Share knowledge with a community of COG Agents Peer to peer networking Benefit from what others know or cause to be known Communicate with other COG Agents and human users Machine and human processable knowledge representations Persistent Knowledge Bases Real Time Response Mixed Initiative Dialog

Augmented Intelligence: Creation of Personal Autonomic Cognition Amplifiers Maxim is “Activity Monitoring” enables “Intention Awareness” enables “Cognition Amplifiers” enables “80%+ Automation” Intention Awareness can be Inferred from Prior Activity Patterns of the User (and Others with Similar Profile) Intention Awareness enables Creation of Personal Autonomic Cognition Amplifiers (Cogs) Cogs are created by continuously monitoring multimodal sensors on modern smart phones, and discovering and interpreting user activities and intentions thru non-intrusive learning technologies. The Activity Monitoring System non-intrusively adapts and creates personalized Cogs using Cog Templates from the Platform providers and Cog developers Cogs enable Automation of Much of the Human Work and Daily Personal Activities Than One-size-fits-all Apps

COG for Everyone on the Planet: Cognition Amplifiers and Guardian Angels Cognition Amplifiers and Guardian Angels are two families of intelligent Agents that help with scarcity of attention problem A Cognition Amplifier is a Personal Enduring Autonomic Intelligent Agent that anticipates what you want to do and helps you to do it with less effort Buying and selling: Transact with multiple providers Email: Filter spam, understand and respond to actionable email News: Based on topic preferences, novelty, collaborative filtering Banking: Monitor bank account, Credit Cards, Pay Bills Travel: Flights, hotel, schedule disruptions, cancellations A Guardian Angel is a Personal Enduring Autonomic Intelligent Agent that Discovers And Warns You About Unanticipated Events That Could Impact Your Safety, Security, and Happiness Just-in-Time Warnings: Hurricanes, Earthquakes, Extreme Weather Accident Alerts and Rerouting; Transport Strikes Scarcity of Essential Resources: Food, Energy, Water etc. Thousands of Cogs and Gats as Personal Assistants?  

Architecture of Cogs and Gats Cogs and Gats are Mobile Apps (like a shadow) for Each Person on the Planet Unlike APPs of today, Cogs are Mass-customized to Each Individual Designed to be Non-intrusive, Autonomic, and Device Independent Always On, Always Present and Always Working Always Learning Enduring (life-long) Cogs and Gats Monitor, Analyze and Learn From Experience; Learn From Own Experience And Experience of Others And share knowledge with a community of Cogs and Gats Automated Discovery of Data and Information Sources Data, suitably anonymized, can be used to learn appropriate responses for every possible situation by Learning preferences by observing user choices, Learning by task similarity and user similarity, Learning by error correction and Simply learning thru clarification dialog ( does that mean yes? Would you care to define it?).

Opt-in “Waze” like Models COG Policy Issues Privacy Individual Other participants Legal Can be subpoena-ed Security Information falling into wrong hands Opt-in “Waze” like Models

Conclusion Pay Attention to Human Strengths, Limitations and Needs NEVER Intrusive Ads on the right or bottom, no animation Abolish Dialog Boxes - Replace with Notifications! Replace “Live Tiles” with WIWTBDIWC Glasses and Watches To Invisible Wearables Scarcity of Attention Implies Creation of Tools for Minimization of Human Effort Use Cogs Autocomplete Tasks, not Just Text Use Gats to Help with Unpredictable Events Avoid Redesigns that Require Re-Learning Ribbons in Word Win 7 to Win 8 $10 Rule: Are You Willing to Pay Every Customer $10 Every Time You Radically Change Look And Feel Of Your Software Product? One Size Fits All Software  Personalized Modular Software Personalize and Mass-customize everything