1. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 1 Write and Publish an IR Paper ChengXiang Zhai Department of Computer Science Graduate School of Library & Information Science Institute for Genomic Biology, Statistics University of Illinois, Urbana-Champaign http://www-faculty.cs.uiuc.edu/~czhai, czhai@cs.uiuc.edu

2. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 2 When to Write a Paper? Survey/Review paper: –An emerging field or topic has appeared (i.e., a hot topic) but no survey is available, or sufficient new development has occurred such that existing surveys are out of date –You’ve read and digested enough papers about the topic Original research paper: when you have sufficient results to draw an interesting conclusion or answer an interesting research question, i.e., you’ve got a basic story to tell: –A new problem, a solution, and results showing how good the solution is –An old problem, a new solution, and results showing advantage(s) of the new solution over the old ones –An old problem, many old solutions, and results showing an understanding of their relative performance –In general, a research question and an answer

3. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 3 Before you write any paper, be clear about the targeted readers

4. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 4 Typical Structure of a Survey Paper Introduction: –Motivation for the survey An emerging field/topic, but no survey available Surveys exist, but they are out of date (e.g., due to new development in a field/topic) –Scope of the survey Background (if necessary) Conceptual framework ( based on synthesis of the literature) –Define basic concepts, terminology, etc –Give a big picture of the topic so that your survey is coherent

5. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 5 Typical Structure of a Survey Paper (cont.) Systematic review of existing work –It’s very important that you have some clear structure for this part The structure is usually your conceptual framework, or other meaningful structures (e.g., by time or some way to classify all the work) –Be critical! Add your opinions about the work surveyed –Don’t treat every work equally; elaborate on some representative work and simply give pointers to other work Summary –Summarize the progress and the state of the art –Give recommendations if any (e.g., for practitioners) –Outlook (remaining challenges, future directions) References

6. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 6 Typical Structure of a Research Paper 1. Introduction –Background discussion to motivate your problem –Define your problem –Argue why it’s important to solve the problem –Identify knowledge gap in existing work or point out deficiency of existing answers/solutions –Summarize your contributions –Briefly mention potential impact Tips: –Start with sentences understandable to almost everyone –Tell the story at a high-level so that the entire introduction is understandable to people with no/little technical background in the topic –Use examples if possible

7. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 7 Typical Structure of a Research Paper (cont.) 2. Previous/Related work –Sometimes this part is included in the introduction or appears later –Previous work = work that you extend (readers must be familiar with it to understand your contribution) –Related work = work related to your work (readers can until later in the paper to know about it) Tips: –Make sure not to miss important related work –Always safer to include more related work –Discuss the existing work and its connection to your work Your work extends … Your work is similar to … but differs in that … Your work represents an alternative way of … –Whenever possible, explicitly discuss your contribution in the context of existing work

8. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 8 Typical Structure of a Research Paper (cont.) 3. Problem definition/formulation –Clearly define your problem If it’s a new problem, discuss its relation to existing related problems If it’s an old problem, cite the previous work –Justify why you define the problem in this way –Discuss challenges in solving the problem Tips: –Give both an informal description and a formal description if possible –Make sure that you mention any assumption you make when defining the problem (e.g., your focus may be on studying the problem in certain conditions)

9. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 9 Typical Structure of a Research Paper (cont.) 4. Overview of the solution(s) (can be merged with the next part) –Give a high-level information description of the proposed solutions or solutions you study –Use examples if possible 5. Specific components of your solution(s) –Be precise (formal description helps) –Use intuitive descriptions to help people understand it Tips: –make sure that you organize this part so that it’s understandable to people with various backgrounds –Don’t just throw in formulas; include high-level intuitive desciptions whenever possible

10. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 10 Typical Structure of a Research Paper (cont.) 6. Experiment design: make sure you justify it –Data set –Measures –Experiment procedure Tips: –Given enough details so that people can reproduce your experiments –Discuss limitation/bias if any, and discuss its potential influence on your study

11. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 11 Typical Structure of a Research Paper (cont.) 7. Result analysis: –Organized based on research questions to be answered or hypotheses tested –Be comprehensive, but focus on the major conclusions –Include “standard” components Baseline comparison Individual component analysis Parameter sensitivity analysis Individual query analysis Significance test –Discuss the influence of any bias or limitation Tips –Don’t leave any question unanswered (try to provide an explanation for all the observed results) –Discuss your findings in the context of existing work if possible Similar observations have also been made in … This is in contrast to … observed in … One explanation is ….

12. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 12 Typical Structure of a Research Paper (cont.) 8. Conclusions and future work –Summarize your contributions –Discuss its potential impact –Discuss its limitation and point out directions for future work 9. References

13. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 13 Where to Publish IR Papers Core IR conferences: –ACM SIGIR, ACM CIKM –ECIR, AIRS Core IR journals –ACM TOIS, IRJ –IPM, JASIS Web Applications –WWW, WSDM Other related conferences –Natural Language Processing: HLT, ACL, NAACL, COLING, EMNLP –Machine Learning: ICML, NIPS –Data Mining: KDD, ICDM –Databases: SIGMOD, VLDB, ICDE …

14. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 14 What an IR reviewer often looks for Most important factors: –Realistic setup of a retrieval problem What kind of users would benefit from your research? –Solid evaluation of methods Truly state of the art baseline Careful selection of data sets –Use as many representative data sets as possible –Always use a standard data set (e.g., TREC) if possible Careful definition of measures Unbiased experiment procedure General factors: –Quality of argument, novelty, writing, … –Avoid all kinds of careless mistakes! (If you aren’t careful about writing, it’s possible you aren’t careful about your experiments either.)

15. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 15 After You Get Reviews Back Carefully classify comments into: –Unreasonable comments (e.g., misunderstanding): Try to improve the clarity of your writing –Reasonable comments Constructive: easy to implement Non-constructive: think about it, either argue the other way or mention weakness of your work in the paper If paper is accepted –Take the last chance to polish the paper as much as you can –You’ll regret if later you discover an inaccurate statement of a typo in your published paper If paper is rejected –Digest comments and try to improve the research work and the paper –Run more experiments if necessary –Don’t try to please reviewers (the next reviewer might say something opposite); instead use your own judgments and use their comments to help improve your judgments –Reposition the paper if necessary (again, don’t reposition it just because a reviewer rejected your original positioning)

16. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 16 Final Tips Start with the core messages you want to convey in the paper and expand your paper by following the core story Try to convey the core messages at different levels so that people with different knowledge background can all get them Try to write a review of your paper yourself, commenting on its originality, technical soundness, significance, evaluation, etc, and then revise the paper if needed Check out reviewer’s instructions, e.g., the following: http://nips07.stanford.edu/nips07reviewers.html (not necessarily matching your conference, but should share a lot of common requirements) http://nips07.stanford.edu/nips07reviewers.html Try to polish English as much as you can

17. 2008 © ChengXiang Zhai Dragon Star Lecture at Beijing University, June 21-30, 2008 17 Good Luck!