1. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 1 Frame an IR Research Problem and Form Hypotheses ChengXiang Zhai Department of Computer Science Graduate School of Library & Information Science Institute for Genomic Biology, Statistics University of Illinois, Urbana-Champaign http://www-faculty.cs.uiuc.edu/~czhai, czhai@cs.uiuc.edu

2. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 2 General Steps to Define a Research Problem Generate and Test Raise a question Novelty test: Figure out to what extent we know how to answer the question –There’s already an answer to it: Is the answer good enough? Yes: not interesting, but can you make the question more challenging? No: your research problem is how to get a better answer to the raised question –No obvious answer: you’ve got an interesting problem to work on Tractability test: Figure out whether the raised question can be answered –I can see a way to answer it or potentially answer it: you’ve got a solvable problem –I can’t easily see a way to answer it: Is it because the question is too hard or you’ve not worked hard enough? Try to reframe the problem to make it easier Evaluation test: Can you obtain a data set and define measures to test solutions/answers? –Yes: you’ve got a clearly defined problem to work on –No: can you think of anyway to indirectly test the solutions/answers? Can you reframe the problem to fit the data? Every time you reframe a problem, try to do all the three tests again.

3. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 3 Rigorously Define Your Research Problem Exploratory: what is the scope of exploration? What is the goal of exploration? Can you rigorously answer these questions? Descriptive: what does it look like? How does it work? Can you formally define a principle? Evaluative: can you clearly state the assumptions about data collection? Can you rigorously define measures? Explanatory: how can you rigorously verify a cause? Predictive: can you rigorously define what prediction is to be made?

4. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 4 Frame a New Computation Task Define basic concepts Specify the input Specify the output Specify any preferences or constraints

5. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 5 Map of IR Applications Web pages News articles Email messages Literature Organization docs Legal docs/Patents Medical records Customer complaint letter/transcripts … Kids Peking Univ. community LawyersScientists SearchBrowsingAlertMining Task/Decision support Customer Service People Email management + automatic reply “Google Kids” Legal Info Systems Literature Assistant Intranet Search Local Web Service Blog articles Online Shoppers ?

6. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 6 From a new application to a clearly defined research problem Try to picture a new system, thus clarify what new functionality is to be provided and what benefit you’ll bring to a user Among all the system modules, which are easy to build and which are challenging? Pick a challenge and try to formalize the challenge –What exactly would be the input? –What exactly would be the output? Is this challenge really a new challenge (not immediately clear how to solve it)? –Yes, your research problem is how to solve this new problem –No, it can be reduced to some known challenge: are existing methods sufficient? Yes, not a good problem to work on No, your research problem is how to extend/adapt existing methods to solve your new challenge Tuning the problem

7. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 7 Tuning the Problem Level of Challenges Impact/Usefulness Known Unknown Make a hard problem easier Make an easy problem harder Increase impact (more general)

8. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 8 Examples of Problem Formulation Risk minimization framework Study of smoothing Axiomatic retrieval framework Comparative Text Mining Contextual PLSA Opinion Integration

9. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 9 Form Research Hypotheses Typical hypotheses in IR: –Hypothesis about user characteristics (tested with user studies or user- log analysis, e.g., clickthrough bias) –Hypothesis about data characteristics (tested with fitting actual data, e.g., Zipf’s law) –Hypothesis about methods (tested with experiments): Method A works (or doesn’t work) for task B under condition C by measure D (feasibility) Method A performs better than method A’ for task B under condition C by measure D (comparative) Introduce baselines naturally lead to hypotheses Carefully study existing literature to figure our where exactly you can make a new contribution (what do you want others to cite your work as?) The more specialized a hypothesis is, the more likely it’s new, but a narrow hypothesis has lower impact than a general one, so try to generalize as much as you can to increase impact But avoid over-generalize (must be supported by your experiments) Tuning hypotheses (next lecture)

10. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 10 Next Lecture (June 26): Test/Refine Hypothese