1. 1INFO4990 Research Methods, s2 2008 INFO4990 Research Methods Bing Bing Zhou http://www.cs.usyd.edu.au/~info4990/ Lecture based in part on materials by Alan Fekete, Mary Lou Maher, Joseph Davis, Irena Koprinska and others Literature Review and Literature Search

2. 2INFO4990 Research Methods, s2 2008 Outline Literature review Literature review why do you need to review the literature why do you need to review the literature goals of literature reviews goals of literature reviews How to search How to search Finding relevant papers Finding relevant papers Search practicalities Search practicalities How to read the literature How to read the literature

3. INFO4990 Research Methods, s2 2008 3 “If I have seen further, it is only because I have stood on the shoulders of giants”. Attributed to Sir Isaac Newton

4. 4INFO4990 Research Methods, s2 2008 Why do you Need to Review the Literature? To help you find a research question To help you find a research question What is the current state of the art? What is the current state of the art? Where are the gaps or the next steps? Where are the gaps or the next steps? To help you establish the importance of the research question To help you establish the importance of the research question Why is this problem important and worth researching? Why is this problem important and worth researching? To establish your work as part of the knowledge base of your research community, e.g. To establish your work as part of the knowledge base of your research community, e.g. “[Cite] defined the following concept, about which we prove …” “[Cite] defined the following concept, about which we prove …” “[Cite1, cite2, cite3] have all worked on systems like this” “[Cite1, cite2, cite3] have all worked on systems like this” “our performance is better than that of [Cite]” “our performance is better than that of [Cite]” To solve your problem To solve your problem E.g. combining ideas from several papers E.g. combining ideas from several papers

5. 5INFO4990 Research Methods, s2 2008 Goals of the Literature Review Demonstrate that you know the field Demonstrate that you know the field Your research question follows from the literature review Your research question follows from the literature review Justifies your research, provides the rationale for the research Justifies your research, provides the rationale for the research how does your work connect to previous work how does your work connect to previous work how does your work differ from previous work how does your work differ from previous work Allows you to establish the conceptual framework and methodological focus Allows you to establish the conceptual framework and methodological focus

6. 6INFO4990 Research Methods, s2 2008 Literature Review Not just a summary of related papers but a critical examination of these papers Not just a summary of related papers but a critical examination of these papers What are their strengths and limitations? What are their strengths and limitations? How exactly do they relate to your question? How exactly do they relate to your question? Should be coherent and structured Should be coherent and structured Learning from examples Learning from examples Ask your supervisor to suggest good previous thesis (Honours/Masters/ PhD) and research papers (journal/conference) Ask your supervisor to suggest good previous thesis (Honours/Masters/ PhD) and research papers (journal/conference) Many available on-line Many available on-line Discuss the literature review section and the overall thesis/paper structure together! Discuss the literature review section and the overall thesis/paper structure together!

7. 7INFO4990 Research Methods, s2 2008 How to Find Relevant Papers Papers given by your supervisor Papers given by your supervisor In many cases your research question will be defined as modification/extension of existing work In many cases your research question will be defined as modification/extension of existing work Papers you found by searching and following up Papers you found by searching and following up Based on terms of interest Based on terms of interest Based on links from other work you find Based on links from other work you find Reference chains Reference chains Papers listed as references in an important paper are also worth examining Papers listed as references in an important paper are also worth examining Community links – important people, journals, conferences Community links – important people, journals, conferences

8. 8INFO4990 Research Methods, s2 2008 Communities of Interest Your work should be targeted at a community Your work should be targeted at a community It must be explained in relation to work in that community It must be explained in relation to work in that community Find who are the important people in the community and where their work is published Find who are the important people in the community and where their work is published Find their websites using Google Find their websites using Google What else they have written? What else they have written? Who are their students? Who are their students? Do they continue to work on the same topic? Do they continue to work on the same topic? What are their current projects What are their current projects

9. 9INFO4990 Research Methods, s2 2008 Searching for Papers - Some Practicalities Given a reference to a paper, you need to get the actual content Given a reference to a paper, you need to get the actual content Look online Look online most of the literature is available as pdf even very old papers have been scanned most of the literature is available as pdf even very old papers have been scanned ACM Digital Library – full access via the library’s home page ACM Digital Library – full access via the library’s home page IEEE Digital Library – full access via the library’s home page IEEE Digital Library – full access via the library’s home page Publisher’s site, e.g. Springer, Elsevier, etc. – hundreds of journals accessible via the library’s home page Publisher’s site, e.g. Springer, Elsevier, etc. – hundreds of journals accessible via the library’s home page Google Scholar Google Scholar Citeseer Citeseer Just using Google Just using Google with the author’s name and a few rare words from the title with the author’s name and a few rare words from the title

10. 10INFO4990 Research Methods, s2 2008 Effective Search We are all used to Google-style search We are all used to Google-style search List words of interest List words of interest Make sensible choices for your search Make sensible choices for your search prefer words that are not very common in general English prefer words that are not very common in general English not much point searching for “design” or “information” or “relation” but “congestion” and “heterogeneity” will work not much point searching for “design” or “information” or “relation” but “congestion” and “heterogeneity” will work often add a term for the field as a whole, e.g. “database” or “network” often add a term for the field as a whole, e.g. “database” or “network” Start with a few targeted words and add more if too much junk is returned Start with a few targeted words and add more if too much junk is returned

11. 11INFO4990 Research Methods, s2 2008 Effective Search (2) Most search tools offer a variety of features, especially important when words of interest are common Most search tools offer a variety of features, especially important when words of interest are common http://www.google.com.au/intl/en/help/basics.html http://www.google.com.au/intl/en/help/basics.html http://www.google.com.au/intl/en/help/basics.html http://www.google.com.au/intl/en/help/refinesearch.html http://www.google.com.au/intl/en/help/refinesearch.html AND (implicit in Google) AND (implicit in Google) OR OR Exact phrase match (Goggle: “words in exact order”) Exact phrase match (Goggle: “words in exact order”) Allow synonyms (Google: ~tern) Allow synonyms (Google: ~tern) Exclude terms (Google: -term) - useful for terms with more than one meaning. Exclude terms (Google: -term) - useful for terms with more than one meaning. Include common terms (Google: +term) – common terms are excluded by default, e.g. “how’, where’ Include common terms (Google: +term) – common terms are excluded by default, e.g. “how’, where’

12. 12INFO4990 Research Methods, s2 2008 Effective Search (3) Topic-Specific Searches in Google Topic-Specific Searches in Google Google Book Search - Search the full text of books Google Book Search - Search the full text of books Google Book Search Google Book Search Google Code Search - Search public source code Google Code Search - Search public source code Google Code Search Google Code Search Google Scholar - Search scholarly papers Google News archive search - Search historical news Google Scholar - Search scholarly papers Google News archive search - Search historical news Google Scholar Google News archive search Google Scholar Google News archive search

13. 13INFO4990 Research Methods, s2 2008 How to Reference Web Documents Some publications exist only on the web Some publications exist only on the web No formal journal or conference publication No formal journal or conference publication May be TR May be TR Reference them by giving the URL used and the date on which it was accessed Reference them by giving the URL used and the date on which it was accessed web information changes web information changes Also, be careful when getting a version from the author’s web page as it may be posted in preliminary not final version Also, be careful when getting a version from the author’s web page as it may be posted in preliminary not final version

14. 14INFO4990 Research Methods, s2 2008 Guide to Research Literature Types of publications Types of publications conference and workshop papers conference and workshop papers journal papers journal papers technical reports technical reports monographs monographs

15. 15INFO4990 Research Methods, s2 2008 Conference Papers Call for papers - ~1 year before meeting Call for papers - ~1 year before meeting Paper submission - ~4-8 months before meeting Paper submission - ~4-8 months before meeting Page limit e.g. 8 pages Page limit e.g. 8 pages Details often omitted (proofs, design technicalities) Details often omitted (proofs, design technicalities) Program Committee reviews the papers Program Committee reviews the papers Criteria: relevance, significance, originality, soundness, readability Criteria: relevance, significance, originality, soundness, readability Final version for proceedings due ~3 months before meeting Final version for proceedings due ~3 months before meeting revise by author in light of reviews revise by author in light of reviews but not checked again but not checked again Annual or bi-annual conferences Annual or bi-annual conferences

16. 16INFO4990 Research Methods, s2 2008 Selection Process Typically 3 reviewers Typically 3 reviewers Acceptance rate – varies Acceptance rate – varies Some 10-15%, others 50% Some 10-15%, others 50% Some review “blind” (author details not shown to reviewers), others do not Some review “blind” (author details not shown to reviewers), others do not Ask your supervisor for guidance about which are the reliable and important conferences in your field! Ask your supervisor for guidance about which are the reliable and important conferences in your field!

17. 17INFO4990 Research Methods, s2 2008 Workshop Papers A workshop is typically a smaller meeting than a conference A workshop is typically a smaller meeting than a conference Sometimes workshop papers are just like conference papers Sometimes workshop papers are just like conference papers Other workshops are more preliminary Other workshops are more preliminary can publish a position paper (draft of an idea without evidence, or proposal for future work) can publish a position paper (draft of an idea without evidence, or proposal for future work) less rigorously reviewed, the goal is mainly to allow the community to meet less rigorously reviewed, the goal is mainly to allow the community to meet

18. 18INFO4990 Research Methods, s2 2008 Journal Article Typically longer than a conference paper Typically longer than a conference paper Often based on a conference paper with additions, corrections and improvements Often based on a conference paper with additions, corrections and improvements Refereed by Refereed by at least 3 reviewers, experts in the field at least 3 reviewers, experts in the field they spend months on the paper checking details, etc. they spend months on the paper checking details, etc. Decisions: accepted, accepted with minor revisions, major revisions and resubmission, rejected Decisions: accepted, accepted with minor revisions, major revisions and resubmission, rejected Revisions, refereed again Revisions, refereed again Accepted, published after several months (journal issues have limited capacity) Accepted, published after several months (journal issues have limited capacity) Time from submission to publication varies, typically 1-1.5 years but may be 3-4 years Time from submission to publication varies, typically 1-1.5 years but may be 3-4 years

19. 19INFO4990 Research Methods, s2 2008 Standard of Journals Many journals in each area with different standards Many journals in each area with different standards Typically IEEE Transactions and ACM Communications are some of the top-ranked journals Typically IEEE Transactions and ACM Communications are some of the top-ranked journals Not all IEEE Trans. and ACM Comm. are top journals Not all IEEE Trans. and ACM Comm. are top journals Not all papers published in top journals are good papers Not all papers published in top journals are good papers Ask you supervisor which journals are the top-ranked and most important in your area! Ask you supervisor which journals are the top-ranked and most important in your area!

20. 20INFO4990 Research Methods, s2 2008 Technical Report Issued by the author’s department, with a number and date Issued by the author’s department, with a number and date May be based on a conference paper May be based on a conference paper Longer, includes all the boring details that are omitted from the conference paper due to space limitations Longer, includes all the boring details that are omitted from the conference paper due to space limitations Used to establish priority Used to establish priority E.g. produce TR before submitting to conference or journal + conference and journal papers may get rejected E.g. produce TR before submitting to conference or journal + conference and journal papers may get rejected

21. 21INFO4990 Research Methods, s2 2008 PhD or MSc Thesis Very extensive account Very extensive account Show much of the research process Show much of the research process Extensive survey of the literature Extensive survey of the literature Very complete evaluation of the work Very complete evaluation of the work The goal is to establish that the author is ready to become independent researcher The goal is to establish that the author is ready to become independent researcher i.e. PhD and MSc provide research training i.e. PhD and MSc provide research training Typically checked by 2 or 3 reviewers Typically checked by 2 or 3 reviewers

22. 22INFO4990 Research Methods, s2 2008 Monograph A collection of selected papers from a conference or workshop A collection of selected papers from a conference or workshop A bit more checking than for the conference/workshop A bit more checking than for the conference/workshop An author can offer a coherent and unified account of a whole research topic An author can offer a coherent and unified account of a whole research topic often combines their own results with other people’s often combines their own results with other people’s Revisits several papers using unified notation, better exposition, better literature review, etc. Revisits several papers using unified notation, better exposition, better literature review, etc. Publisher may get reviewers but their focus is “will it sell” not “is it correct” Publisher may get reviewers but their focus is “will it sell” not “is it correct”

23. 23INFO4990 Research Methods, s2 2008 Warnings There's a whole bunch of conferences/journals in your community. There's a whole bunch of conferences/journals in your community. You could spend all your time reading them. Fortunately, only a few are worth looking at. You could spend all your time reading them. Fortunately, only a few are worth looking at. Quality of conferences and journals varies, and this is reflected in the checking of the papers Quality of conferences and journals varies, and this is reflected in the checking of the papers Read papers with a critical eye! Read papers with a critical eye! Some communities are very clique-dominated Some communities are very clique-dominated Unpopular opinions are not welcome Unpopular opinions are not welcome Clique leaders can publish anything, even half-baked ideas without evidence Clique leaders can publish anything, even half-baked ideas without evidence

24. 24INFO4990 Research Methods, s2 2008 Fake Conferences and Random Papers http://pdos.csail.mit.edu/scigen/ http://pdos.csail.mit.edu/scigen/ http://pdos.csail.mit.edu/scigen/ A random paper accepted to a journal? A random paper accepted to a journal?

25. 25INFO4990 Research Methods, s2 2008 The Research Community A community has conferences and journals of high prestige which they read and publish in A community has conferences and journals of high prestige which they read and publish in They meet often, and each knows (more or less) what others are doing They meet often, and each knows (more or less) what others are doing You must place your work in the context of a community You must place your work in the context of a community

26. 26INFO4990 Research Methods, s2 2008 Quality Metrics How important is an article? How influential is an author? How important is an article? How influential is an author? Based on citation analysis - number of times a paper or author is cited Based on citation analysis - number of times a paper or author is cited How to calculate citations – Google Scholar + other software How to calculate citations – Google Scholar + other software Assumption: important authors and articles are cited more often than the others Assumption: important authors and articles are cited more often than the others Increasingly used by governments, funding bodies, promotion committees to evaluate the quality of author’s work Increasingly used by governments, funding bodies, promotion committees to evaluate the quality of author’s work Some drawbacks Some drawbacks Citing errors – authors with the same names are not separated Citing errors – authors with the same names are not separated Cliques (friends, colleagues) cite each other in turn to build their citation index Cliques (friends, colleagues) cite each other in turn to build their citation index Negative citations are included (citations to incorrect results) Negative citations are included (citations to incorrect results)

27. 27INFO4990 Research Methods, s2 2008 ISI Citation Database Very popular, established in 1960, contains >40million records, contains Very popular, established in 1960, contains >40million records, contains Arts and Humanities Citation Index (A&HCI) Arts and Humanities Citation Index (A&HCI) Science Citation Index (SCI) Science Citation Index (SCI) Social Sciences Citation Index (SSCI) Social Sciences Citation Index (SSCI) However However it doesn’t index a large number of journals it doesn’t index a large number of journals ignores open-access journals ignores open-access journals doesn’t index conferences doesn’t index conferences “Read the Rise and Rise of Citation Analysis” by L. Meho!

28. 28INFO4990 Research Methods, s2 2008 Journal’s Impact Factor Journal impact factors Journal impact factors Used to determine the importance of a journal Used to determine the importance of a journal E.g. journal impact factor for 2007 = E.g. journal impact factor for 2007 = # citations in 2007 to articles published in the journal in 2005-6 # citations in 2007 to articles published in the journal in 2005-6 = ------------------------------------------------------------------------------- # articles published in the journal in 2005-6 # articles published in the journal in 2005-6

29. 29INFO4990 Research Methods, s2 2008 CORE’s ratings Computing Research and Education Association of Australasia (CORE) Computing Research and Education Association of Australasia (CORE) Australia and New Zealand Australia and New Zealand Ranking of journals and conferences in CS – not finalised Ranking of journals and conferences in CS – not finalisedhttp://www.core.edu.au/

30. 30INFO4990 Research Methods, s2 2008 How to Read the Literature Reading papers is a skill that takes practice. Reading papers is a skill that takes practice. You can't afford to read in full all the papers that come to you. You can't afford to read in full all the papers that come to you. Normally there are three phases to reading one. Normally there are three phases to reading one. The first is to see if there's anything of interest in it at all. Abstracts, the table of contents, conclusion section, and introduction are good places to look. The first is to see if there's anything of interest in it at all. Abstracts, the table of contents, conclusion section, and introduction are good places to look. Once you've figured out what in general the paper is about and what the claimed contribution is, you can decide whether or not to go on to the second phase, which is to find the part of the paper that has the good stuff. Once you've figured out what in general the paper is about and what the claimed contribution is, you can decide whether or not to go on to the second phase, which is to find the part of the paper that has the good stuff. Finally, you may go back and read the whole paper through if it seems worthwhile. Finally, you may go back and read the whole paper through if it seems worthwhile.

31. 31INFO4990 Research Methods, s2 2008 How to Read the Literature(2) With a critical eye With a critical eye Take notes Take notes What is the contribution? What is the contribution? What is the evidence? What is the research method used? What is the evidence? What is the research method used? How does this contribution relate to the previous work? How does this contribution relate to the previous work? What are the important references cited in the paper? (follow them up) What are the important references cited in the paper? (follow them up) Do you have any doubts, questions, ideas? Do you have any doubts, questions, ideas? Keep reading the important journals and conference papers in your field Keep reading the important journals and conference papers in your field IT and Eng research ages quickly IT and Eng research ages quickly Maintain annotated bibliography from the start Maintain annotated bibliography from the start

32. 32INFO4990 Research Methods, s2 2008 How to Read the Literature(3) Don’t rely on second hand summaries! Go to the original source if you can! Don’t rely on second hand summaries! Go to the original source if you can! Get attributions right in your own writing: don’t just accept citations from other work, even with full reference! Get attributions right in your own writing: don’t just accept citations from other work, even with full reference! When reading a well written paper When reading a well written paper What makes this paper easy to read? What makes this paper easy to read? What level of detail is provided? What level of detail is provided? What examples are used to demonstrate important concepts? What examples are used to demonstrate important concepts? What questions are left unanswered (future work)? What questions are left unanswered (future work)?

33. 33INFO4990 Research Methods, s2 2008 Organising the Literature Isolate issues and highlight the findings and contributions that are central to your research Isolate issues and highlight the findings and contributions that are central to your research Group together papers that deal with a common or related theme or issue Group together papers that deal with a common or related theme or issue Organise the materials with diagrams, tables, concept maps Organise the materials with diagrams, tables, concept maps Try out different structures for organising; they should be most relevant to the goals of your research Try out different structures for organising; they should be most relevant to the goals of your research Chronological order is not particularly useful but citation chains are useful Chronological order is not particularly useful but citation chains are useful Warning Warning papers often don’t use common terminology, or focus on common issues, or explain relationships fairly, clarifying these aspects is a key contribution you can make papers often don’t use common terminology, or focus on common issues, or explain relationships fairly, clarifying these aspects is a key contribution you can make