1. Henry Lieberman MIT Media Lab User Interface Issues for Agents & Adaptive Software Henry Lieberman Media Laboratory Massachusetts Institute of Technology Cambridge, MA 02139 USA lieber@media.mit.edu http://www.media.mit.edu/~lieber/

2. Henry Lieberman MIT Media Lab Making it “easy” for machines to understand the Web Machines can’t read HTML intended for humans Well, they can now, somewhat, heuristically The Semantic Web will make it “easy” for machines to understand knowledge on the Web Yeah, but only if people know what knowledge to represent and how to encode it Yeah, but only if people know how to make agents that use the knowledge

3. Henry Lieberman MIT Media Lab User interface problems User interface problem for the user/Web author Increasing number of knowledge engineers by 10000s of times Will user encode everything twice? User interface problem for agent developer Whole environment too !?#&*@ complex Can’t encode procedural knowledge Can’t debug

4. Henry Lieberman MIT Media Lab User interface for Web developers is becoming unmanageable How many languages do you need to know to program the Web? W3C has 19 languages on its home page XML+extensions, Javascript, Java, CGI+Perl, Php… No defined interoperability Unreasonably difficult to debug!

5. Henry Lieberman MIT Media Lab Static semantics of the Web Structure of Web pages and links Including meta-data [RDF, DAML+OIL, etc.]

6. Henry Lieberman MIT Media Lab Dynamic semantics of the Web The user browses through Web pages The user’s interests change Programs “crawl” through Web pages Web pages do stuff Web pages change

7. Henry Lieberman MIT Media Lab Dynamic semantics of the Web Water: Encoding dynamic semantics in Web “pages” User interfaces for understanding dynamic behavior (Program execution or Inference) Instrumentation & Localization Control of level of detail History & Reversibility

8. Henry Lieberman MIT Media Lab Water (née Glue) - Fry, Plusch, Lieberman Embeds procedures in Web pages Integrates content, code, program data XML-compatible syntax Prototype object system Integrated development environment, debugger

9. Henry Lieberman MIT Media Lab Water

10. Henry Lieberman MIT Media Lab ZStep Keeps complete history of computation Reversible Control of level of detail Multiple, synchronized views of process state User-interface and implementation-level views

11. Henry Lieberman MIT Media Lab ZStep

12. Henry Lieberman MIT Media Lab The old “information retrieval” perspective User issues the “perfect query” to a static database System returns the “perfect document” (Keyword1 … KeywordN)?

13. Henry Lieberman MIT Media Lab What’s wrong with the IR view? Users can’t formulate precise queries Empirically: Users do 1-2 word queries, don’t use advanced query languages There is no “best document” in the Web Web keeps growing, changing Real goal: To make the best use of the user’s time Consequence: Web browsing is a real-time activity

14. Henry Lieberman MIT Media Lab Theorem-proving view is similar Query Descriptions and assertions Answer

15. Henry Lieberman MIT Media Lab What’s wrong with the theorem- proving view? Neither users nor programs can formulate precise queries There is no “best answer” in the Web Web keeps growing, changing Real goal: To make the best use of the user’s and agent’s time Consequence: Web browsing and inference is a real-time activity

16. Henry Lieberman MIT Media Lab The new “agents” perspective Web browsing/search/inference should be a cooperative activity between a human user and [one or more] software agents Each participant should do what they do best: Users are good at evaluation Agents are good at search/computation Both are active in real time, communicate

17. Henry Lieberman MIT Media Lab Some semantics will be discovered/computed by agents Not all semantics of the Web will be statically encoded Agents will compute semantics dynamically from natural language [info extraction] Agents will compute semantics from relationships [collab filtering, popularity] How do we integrate statically declared semantics with dynamically computed semantics?

18. Henry Lieberman MIT Media Lab Examples of Web agents Letizia Mindshare Aria Expert Finder Apt Decision

19. Henry Lieberman MIT Media Lab Letizia: An Interface Agent for Assisting Web Browsing Letizia acts as an advance scout for Web browsing: It watches your Web browsing to try to learn what topics you are interested in Formulates “queries” dynamically/incrementally While you are reading a Web page, Letizia searches the neighborhood of the page to discover other pages you might be interested in Does “search” dynamically/incrementally

20. User’s Search [Depth-First]

21. Henry Lieberman MIT Media Lab User’s Search & Letizia’s Search

22. Henry Lieberman MIT Media Lab Advantages of Letizia While you search “wide”, Letizia searches “deep” Letizia uses the time that you spend reading a page to anticipate what you might interested in Letizia filters out “junk” Letizia maintains persistence of interest Letizia is good at discovering serendipitous connections

23. Henry Lieberman MIT Media Lab Mindshare - Van Dyke & Lieberman

24. Henry Lieberman MIT Media Lab Mindshare - Van Dyke & Lieberman Tool for developing collaborative ontologies of Web pages Browser/editor for personalized views of a collaborative ontology Decide which aspects of common ontology to include in your personal ontology Decide which aspects of your personal ontology you wish to contribute to the common ontology

25. Henry Lieberman MIT Media Lab Aria: Annotation and Retrieval Integration Agent - Lieberman Aria = Email/Web editor + Photo database + Agent "Last weekend, I went to Ken and Mary's wedding…"

26. Henry Lieberman MIT Media Lab Aria: Annotation and Retrieval Integration Agent - Henry Lieberman Agent uses the context of the message to infer relevance of photos to text Agent automatically retrieves relevant photos as message is typed Agent automatically annotates photos with relevant text from message Streamlined interaction: No dialog boxes, file names, cut and paste, load and save, typed queries, multiple applications, etc. etc. etc.

27. Henry Lieberman MIT Media Lab Expert Finder - Vivacqua & Lieberman

28. Henry Lieberman MIT Media Lab Expert Finder - Vivacqua & Lieberman Agent to help locate someone who can answer a question Domain of [novice] Java programming Matchmaking in context of user’s and helper’s expertise Agent reads users’ Java programs, relates them to: Java ontology [analyzed from Sun doc] Model of Java expertise [analyzed from “Java in 21 Days”]

29. Henry Lieberman MIT Media Lab Apt Decision - Sybil Shearin Complex decision making for e-commerce Looking for apartment rental Simulates interaction style of real-estate agent with customer Proposals generated from a few criteria User can react to any aspect of proposal at any time System infers preferences

30. Henry Lieberman MIT Media Lab Apt Decision - Sybil Shearin

31. Henry Lieberman MIT Media Lab Berners-Lee, Hendler, Lassila

32. Henry Lieberman MIT Media Lab Sci Am Scenario Local devices Volume Control Prescribed Treatment Providers In-Plan 20-Mile Radius Home Rating SemWeb for small devices Describing appliances Medical SemWeb Directory services Defining concepts Geographic information Domestic applications Web services

33. Henry Lieberman MIT Media Lab Sci Am Scenario Appointment times Location, time preferences Trust Reschedule Miscategorization Details? Personal info spaces User Modeling & adaptation Security Replanning Debugging Explanation

34. Henry Lieberman MIT Media Lab