1. (Spoken) Dialogue and Information Retrieval Antoine Raux Dialogs on Dialogs Group 10/24/2003

2. Outline Interactive Information Retrieval Systems (Belkin et al) Interactive Information Retrieval Systems (Belkin et al) EUREKA: Dialogue-based IR for Low Bandwidth Devices EUREKA: Dialogue-based IR for Low Bandwidth Devices Voice Access to IR Voice Access to IR

3. Cases, Scripts, and Information- Seeking Strategies Belkin, Cool (Rutgers) Stein, Thiel (GMD-IPSI) Belkin, Cool (Rutgers) Stein, Thiel (GMD-IPSI) Long journal article (1995) Long journal article (1995) From the IR community (Expert Systems) From the IR community (Expert Systems)

4. IR as Interaction Traditional IR research focuses on document/query representation and comparison Traditional IR research focuses on document/query representation and comparison Need to focus on the user Need to focus on the user Represent IR as a dialogue between an information seeker and an information provider Represent IR as a dialogue between an information seeker and an information provider

5. Information-Seeking Strategies Represent information-seeking behavior along 4 dimensions: Represent information-seeking behavior along 4 dimensions: Method of Interaction (scanning vs searching) Method of Interaction (scanning vs searching) Goal of Interaction (learning vs selecting) Goal of Interaction (learning vs selecting) Mode of Retrieval (recognition vs specification) Mode of Retrieval (recognition vs specification) Resource Considered (information vs meta-info) Resource Considered (information vs meta-info) Binary values  16 strategies (ISS) Binary values  16 strategies (ISS)

6. Dialogue Structures for Information Seeking Mix of different formalisms: Mix of different formalisms: Recursive state-based schemas (COR) e.g. Request  Promise  Inform  Be contented Recursive state-based schemas (COR) e.g. Request  Promise  Inform  Be contented Scripts: prototypical interaction for each ISS Scripts: prototypical interaction for each ISS Goal trees Goal trees Retrieve Specified Items Specify CharacteristicRecognize Desired Items Offer choiceSelect and Specify

7. Deriving Scripts from Data Case-based approach: problem solving using previously stored solved instances Case-based approach: problem solving using previously stored solved instances Match a sequence of action to a state- based schema Match a sequence of action to a state- based schema Extract goal tree Extract goal tree Identify goal (which ISS?) Identify goal (which ISS?)

8. The MERIT System Theory vs Practice… Theory vs Practice… Graphical interface (not NL dialogue) Graphical interface (not NL dialogue) User does case selection (for eventual case-based reasoning) User does case selection (for eventual case-based reasoning) Example task is relational database (not free text IR): uses form filling (!) Example task is relational database (not free text IR): uses form filling (!)

9. Discussion Contribution to IR: user-centered view, application of many non-IR theories (discourse, CBR) Contribution to IR: user-centered view, application of many non-IR theories (discourse, CBR) BUT: too complicated for the user? BUT: too complicated for the user?

10. Discussion Contribution to Dialogue Systems: difficult task (not often dealt with in DS), CBR (can we learn dialogue structure from data?) Contribution to Dialogue Systems: difficult task (not often dealt with in DS), CBR (can we learn dialogue structure from data?) BUT: lacks a good, unified, practical framework (too many different paradigms applied…) BUT: lacks a good, unified, practical framework (too many different paradigms applied…)

11. Dialogue-based IR: Why? Google-like interface still predominant (despite MERIT) Google-like interface still predominant (despite MERIT) Why? Why? Users receives a lot of information (document titles, summaries) and use it as they want Users receives a lot of information (document titles, summaries) and use it as they want Very simple to learn Very simple to learn Very flexible Very flexible BUT: works on high bandwidth devices BUT: works on high bandwidth devices

12. Dialogue-based IR: Why? For low bandwidth devices (PDA, phone), information-rich interface don’t work For low bandwidth devices (PDA, phone), information-rich interface don’t work Only small pieces of information exchanged at a time Only small pieces of information exchanged at a time System has to select System has to select Less information, more interaction Less information, more interaction

13. EUREKA: Idea Use dialogue to submit queries to a web search engine, browse through the hierarchically clustered results, perform query reformulation/refinement, etc… Use dialogue to submit queries to a web search engine, browse through the hierarchically clustered results, perform query reformulation/refinement, etc…

14. EUREKA: Overview Backend: Vivisimo (through web scraper) Backend: Vivisimo (through web scraper) Dialogue Management: RavenClaw (successor of CMU Communicator) Dialogue Management: RavenClaw (successor of CMU Communicator) Language Understanding: Light Open Vocabulary Parser Language Understanding: Light Open Vocabulary Parser NLG/TTS: template-based & Festival NLG/TTS: template-based & Festival

15. Backend: Vivisimo Available clustering meta-search engine www.vivisimo.com Available clustering meta-search engine www.vivisimo.com www.vivisimo.com Hand-written Perl web scraper (hope Vivisimo doesn’t change their page design by the end of the semester…) Hand-written Perl web scraper (hope Vivisimo doesn’t change their page design by the end of the semester…)

16. LOV Parser Problem: traditional NL parsers require a dictionary  not applicable to open domain IR Problem: traditional NL parsers require a dictionary  not applicable to open domain IR Solution (implemented in C++): Solution (implemented in C++): fix a small number of one-word commands (new_query, open, list_clusters) fix a small number of one-word commands (new_query, open, list_clusters) parse each line as “[command] [arguments]” or “[command]” or “[arguments]” parse each line as “[command] [arguments]” or “[command]” or “[arguments]”

17. Dialogue Management: RavenClaw Hierarchical agent architecture: Hierarchical agent architecture: EUREKA Greet User Prompt Query New Query Open Cluster … Submit Query Get Cluster List Get Doc List Inform Results Close Cluster

18. NLG/TTS Template-based Language Generation (e.g. “I found documents.”) Template-based Language Generation (e.g. “I found documents.”) General purpose Festival voice for TTS General purpose Festival voice for TTS NB: Browsing through lists is not efficient with speech, even for lists of clusters

19. Already Implemented Working prototype Working prototype Commands: Commands: new_query new_query list_clusters, list_documents list_clusters, list_documents open, close (cluster) open, close (cluster) more, back (list of clusters/documents) more, back (list of clusters/documents)

20. Demo

21. Future Work Add more functionalities (query refinement, summarization…) Add more functionalities (query refinement, summarization…) Make clever use of the dialogue (not only command and control + browsing) Make clever use of the dialogue (not only command and control + browsing) System can provide advice to user on search strategies (e.g. “you need to refine the query”) System can provide advice to user on search strategies (e.g. “you need to refine the query”) User and system can negotiate to specify the user’s information need (cf Belkin: overview vs specific document) User and system can negotiate to specify the user’s information need (cf Belkin: overview vs specific document)

22. Future Work/Discussion Advantage of dialogue: more feedback from the user Advantage of dialogue: more feedback from the user How can dialogue improve the efficiency of low bandwidth IR? How can dialogue improve the efficiency of low bandwidth IR? Do we need to tailor IR techniques (e.g. clustering) for dialogue, or even design new techniques? Do we need to tailor IR techniques (e.g. clustering) for dialogue, or even design new techniques?

23. Vocal Access to IR Problem: ASR introduces a lot of erroneous words in a spoken query (for an open domain, speaker independent system) Problem: ASR introduces a lot of erroneous words in a spoken query (for an open domain, speaker independent system) However, in an IR system: access to many text documents to help language modeling… However, in an IR system: access to many text documents to help language modeling…

24. Vocal Access to a Newspaper Archive (Crestani 02) Presents studies for a full voice-controlled IR system Presents studies for a full voice-controlled IR system No dialogue: user query  list of summaries No dialogue: user query  list of summaries Focuses on issues of: Focuses on issues of: TTS: can user make relevance judgments when they hear document descriptions synthesized over the phone? (answer: yes) TTS: can user make relevance judgments when they hear document descriptions synthesized over the phone? (answer: yes) ASR: how does IR perform with recognized queries? ASR: how does IR perform with recognized queries?

25. Using IR Techniques to Deal with Recognition Errors WER does have an impact on precision, although not much variation for WER in 27%-47% WER does have an impact on precision, although not much variation for WER in 27%-47% Relevance feedback: use documents judged relevant by the user as query Relevance feedback: use documents judged relevant by the user as query Use prosodic stress to estimate information content of query terms Use prosodic stress to estimate information content of query terms Include semantically/phonetically close terms in the query Include semantically/phonetically close terms in the query

26. Improving ASR (Fujii et al 02) Fujii et al propose LM adaptation based on the IR corpus: Fujii et al propose LM adaptation based on the IR corpus: Offline “adaptation”: train on the whole corpus Offline “adaptation”: train on the whole corpus Online adaptation: adapt on the top retrieved documents (then reperform ASR and IR) Online adaptation: adapt on the top retrieved documents (then reperform ASR and IR) Good results with offline trained LM (WER < 20%, AP loss of 20-30% from text IR) Good results with offline trained LM (WER < 20%, AP loss of 20-30% from text IR) No evaluation of online adaptation… No evaluation of online adaptation…

27. Vocal Access to IR: Discussion Seems to work ok for some tasks Seems to work ok for some tasks Clever use of IR techniques Clever use of IR techniques BUT queries are not spontaneous nor natural (maybe) BUT queries are not spontaneous nor natural (maybe) LM for Web queries?? LM for Web queries?? What about dialogue? What about dialogue?