5 © Department of Computer Science and Technology, Taiyuan University of Technology Computer English For Computer Major Master Candidates Department of Computer Science and Technology, Taiyuan University of Technology Dr. Peng Xinguang Sep. 07, 2007

5 © Department of Computer Science and Technology, Taiyuan University of Technology How to Read Research Papers in Computer Science and Engineering for the Master Candidates Unit 5

5 © Department of Computer Science and Technology, Taiyuan University of Technology Where are CSE Research Papers Found? CSE research papers may be published as: technical reports, conference papers, journal papers or book chapters (monographs, not teaching materials). Frequently, an author will write a technical report that expands on the information in a conference paper. Results from several conference papers may be combined and expanded into a journal paper. A conference paper may also be expanded later into a book chapter. Finally, Master or PhD dissertations are frequently revised into either journal papers or book chapters.

5 © Department of Computer Science and Technology, Taiyuan University of Technology The timeline is usually: conference paper, technical report, journal paper, book chapter. To get the most recent information, look for conference papers. To get more considered papers with more explanation, look for journal papers. Conference papers and most journal papers are “peer- reviewed”, that is, have been examined by other computer scientists prior to publication. Technical reports are typically not peer-reviewed, but are still excellent sources of detailed information about algorithms and computer systems. Timeline and Difference

5 © Department of Computer Science and Technology, Taiyuan University of Technology Famous References Websites More and more authors post their papers on their personal web sites. If an article is posted on the web, it will probably end up in: Citeseer ( computer documents have been indexed by CiteSeer.IST up to now (Sep. 2007). The Collection of Computer Science Bibliographies ( The collection currently contains more than 2 millions of references (mostly to journal articles, conference papers and technical reports).

5 © Department of Computer Science and Technology, Taiyuan University of Technology Famous References Websites DBLP--Digital Bibliography & Library Project ( The server indexes more than articles and contains several thousand links to home pages of computer scientists (Sep. 2007). Networked Computer Science Technical Reference Library ( Academic Archive On-line ( In DiVA you can find theses, dissertations and other publications in full-text; Other places to look for papers include the websites from your library and journal publishers such as Elsevier and Kluwer, which may even post copies of papers not yet published!

5 © Department of Computer Science and Technology, Taiyuan University of Technology Types of CSE Research Paper There are three basic types of CSE research paper: theoretical, engineering and empirical. A theoretical paper describes a theory or algorithm or provides a mathematical proof for some hypothesis. An engineering paper describes an implementation of an algorithm, or part or all of a computer system or application. Engineering papers are now frequently required to include descriptions of system evaluation. An empirical paper describes an experiment designed to test some hypothesis.

5 © Department of Computer Science and Technology, Taiyuan University of Technology is Research Paper Good? Every year thousands of papers are published by: the ACM(Association for Computing Machinery, ); IEEE(Institute for Electrical and Electronic Engineers, AAAI( American Association for Artificial Intelligence, ACL(Association for Computational Linguistics and other professional societies. You can’t read them all!

5 © Department of Computer Science and Technology, Taiyuan University of Technology 计算机技术类核心期刊 (2004 年版 ) 1 计算机学报 2 软件学报 3 计算机研究与发展 4 计算机科学 5 计算机辅助设计与图形学 学报 6 计算机工程与应用 7 模式识别与人工智能 8 小型微型计算机系统 9 计算机工程 10 计算机应用 11 中国图象图形学报 12 计算机应用研究 13 系统仿真学报 14 计算机集成制造系统 15 中文信息学报 16 微计算机信息 17 数据采集与处理 18 微型机与应用 19 计算机工程与设计 20 计算机应用与软件 21 微型计算机 22 微电子学与计算机

5 © Department of Computer Science and Technology, Taiyuan University of Technology is Research Paper Good? In addition to learning what are good publication in your field and checking citations, you can sometimes tell if a paper is likely to be good by looking for structural clues in the paper itself. Here are some indicators of a good research paper: The problem the paper addresses is clearly stated, both in the abstract and early on in the paper itself. The technical importance and broader impacts of the paper are described.

5 © Department of Computer Science and Technology, Taiyuan University of Technology is Research Paper Good? The paper includes a clear description of the experiment, system or theory the problem addresses. This is usually the second section of the paper. The paper describes and analyzes the results of the work described (either experimental or evaluation results). The authors have some sound ideas for future work. This usually appears at the end of the paper.

5 © Department of Computer Science and Technology, Taiyuan University of Technology is Research Paper Good? Related work is described and cited correctly. You can get an idea of this by looking at the list of references at the end of the paper. If you know that a researcher has been working in an area for a while, that is usually an indicator that the research is sound; however, do not underestimate the contributions of people new to a field or the impact of politics on research.

5 © Department of Computer Science and Technology, Taiyuan University of Technology Reading Papers is Challenging Reading research papers effectively is challenging. These papers are written in a very condensed style because of page limitations and the intended audience, which is assumed to already know the area well. You have to work harder to find the content that you are interested in. Your time is very limited, so you may not have time to read every word of the paper or read it several times to extract all the differences. For all these reasons, reading a research paper can require a special approach.

5 © Department of Computer Science and Technology, Taiyuan University of Technology Skimming the Papers First, skim the paper to check that it has the elements described above (clear problem statement, clear description of method or system, clear presentation of results, appropriate citation of related work). This takes just a few minutes. At the end of this quick skim you should be able to write down the type of the paper (theoretical, engineering or empirical), the area of the paper is about, and the problem the paper addresses.

5 © Department of Computer Science and Technology, Taiyuan University of Technology New Words Questions Reading research papers is partly learning the specific vocabulary of a field. Don't worry! If you approach it step by step, even an impossible- looking paper can be understood. Go through the paper word by word and line by line, underlining or highlighting every word and phrase you don't understand. Don't worry if there are a lot of underlinings; you're still not trying to make sense of the article.

5 © Department of Computer Science and Technology, Taiyuan University of Technology Look up Computer Dictionary Look up simple words and phrases. Often the question is simply vocabulary Understand technical terms. For example, what's an episode pattern, or the false positives or a profile. A computer dictionary is a good place to look for definitions. A English textbook may be a good source, because it give more complete explanations. Your ordinary shelf dictionary is not a good source, because the definitions may not be precise enough or may not reflect the way in which scientists use a word.

5 © Department of Computer Science and Technology, Taiyuan University of Technology Understanding from the Context Get an understanding from the context in which it is used. Often words that are used to describe the procedures used in an experiment can be understood from the context, and may be very specific to the paper you are reading. Of course, you should be careful when deciding that you understand a word from its context, because it might not mean what you think. You would need to understand words describing one of the major concepts of the paper in context and in some depth. Try to deal with all the words and phrases

5 © Department of Computer Science and Technology, Taiyuan University of Technology Awkward English CSE papers are often written in English by non- native speakers of English. Syntactic errors or awkwardness of phrasing do not indicate that the research is bad. You should try distinguish between the writing style and the research itself. If you get confused, you can reread or write down your questions to check in other work by the same authors.

5 © Department of Computer Science and Technology, Taiyuan University of Technology Comprehension 1. What is the problem the paper attempts to address? What is the motivation of the research work? Is there a crisis in the research field that the paper attempts to resolve? Is the research work attempting to overcome the weaknesses of existing approaches? Is an existing research paradigm challenged? In short, what is the gap of the paper?

5 © Department of Computer Science and Technology, Taiyuan University of Technology 2. What are the claimed contributions of the paper? What is new in this paper? A new question is asked? A new understanding of the research problem? A new methodology for solving problems? A new algorithm? A new breed of software tools or systems? Comprehension

5 © Department of Computer Science and Technology, Taiyuan University of Technology Comprehension A new experimental method? A new proof technique? A new formalism or notation? A new evidence to disprove a previously published claim? A new research area? In short, what is original about this paper?

5 © Department of Computer Science and Technology, Taiyuan University of Technology Comprehension 3. How do the authors substantiate their claims? What is the methodology adopted to substantiate the claims? What is the argument of the paper? What are the major theorems? What experiments are conducted? Data analyses? Simulations? Benchmarks? User studies? Case studies? Examples? In short, what makes the claims scientific?

5 © Department of Computer Science and Technology, Taiyuan University of Technology Comprehension 4. What are the conclusions? What have we learned from the paper? Shall the standard practice of the field be changed as a result of the new findings? Is the result generalizable? Can the result be applied to other areas of the field? What are the open problems? In short, what are the lessons one can learn from the paper?

5 © Department of Computer Science and Technology, Taiyuan University of Technology Evaluation 1. Is the research problem significant? Is the work scratching minor itches? Are the authors solving artificial problems? Does the work enable practical applications, deepen understanding, or explore new design space?

5 © Department of Computer Science and Technology, Taiyuan University of Technology Evaluation 2. Are the contributions significant? Is the paper worth reading? Are the authors simply repeating the state of the art? Are there real surprises? Are the authors aware of the relation of their work to existing literature? Is the paper addressing a well-known open problem?

5 © Department of Computer Science and Technology, Taiyuan University of Technology Evaluation 3. Are the claims valid? Have the authors been cutting corners (intentionally or unintentionally)? Has the right theorem been proven? Has problematic experiment setup? Unrealistic, artificial benchmarks? Comparing apples and oranges? Methodological misunderstanding? Do the numbers add up? Are the generalizations valid?

5 © Department of Computer Science and Technology, Taiyuan University of Technology Synthesis Creativity does not arise from the void. Reading research papers is one of the most effective way for generating novel research. When you read a research paper, you should see it as an opportunity for you to come up with new research projects. The following is a list of questions you can ask to help in this direction.

5 © Department of Computer Science and Technology, Taiyuan University of Technology Asking Questions What is the crux of the research problem? What are some alternative approaches to address the problem? What is a better way to substantiate the claim of the authors? What is a good argument against the case made by the authors? How can the research results be improved? Can the research results be applied to another context? What are the open problems raised by this work? Can we do better than the authors?

5 © Department of Computer Science and Technology, Taiyuan University of Technology How to Write Reviews A paper review is a short essay (3~4 pages) reporting what you have learned from reading a research paper. Writing reviews for the papers you have read is a great way to sharpen your paper reading skills. Such a review is typically structured in three sections: summary, evalution, and synthesis.

5 © Department of Computer Science and Technology, Taiyuan University of Technology Summary for Reviews Give a brief summary of the work in your own words. This summary demonstrates your understanding of the paper, and as such it should answer the four questions outlined in comprehension section: research problem, claimed contributions, substantiation and conclusions. It is imperative that you use your own words to summarize the paper. Another way to think of it is that you are writing an alternative, elaborate abstract for the paper.

5 © Department of Computer Science and Technology, Taiyuan University of Technology Evaluation for Reviews Evaluate the work by answering the questions outlined in evaluation section in earlier time. Point out both the strengths and weaknesses of the work. If you are reading a classical paper that has been published for a while, make sure you are reading the paper in the right historical context: What seems to be obvious now might have been ground-breaking then.

5 © Department of Computer Science and Technology, Taiyuan University of Technology Synthesis for Reviews Generate any interesting thoughts you have on the work by consulting the list of questions in synthesis section.

5 © Department of Computer Science and Technology, Taiyuan University of Technology Peer Review Process When a research paper is submitted to a conference or a journal, it will undergo a peer review process, in which the paper is subject to the intense scrutiny( 详细审查 ) of peer researchers. Who review the submitted paper will read the paper in more or less the same way as we outlined in comprehension and evaluation sections, and then they will write up a referee report in a style similar to the paper review discussed in synthesis section. Based on the referee reports, the program chair of a conference or the editor of a journal will then make the decision of whether to accept the paper.