1. 1INFO5993 Research Methods, s1 2009 INFO5993 Research Methods Kalina Yacef http://www.cs.usyd.edu.au/~info5993/ Lecture based in part on materials by Alan Fekete, Mary Lou Maher, Joseph Davis, Irena Koprinska and others Research – Components and Process Research Publications – Types and Quality Metrics

2. 2INFO5993 Research Methods, s1 2009 Outline Administrative matters Administrative matters Research Research Definition, key components, process Definition, key components, process Finding a research question Finding a research question Guide to research literature Guide to research literature types of publications and how are they produced types of publications and how are they produced Quality metrics – how to measure research impact? Quality metrics – how to measure research impact?

3. 3INFO5993 Research Methods, s1 2009 Administravia Course web page: http://www.cs.usyd.edu.au/~info5993/ Course web page: http://www.cs.usyd.edu.au/~info5993/ 2 hours lectures, 3-5pm on Mondays 2 hours lectures, 3-5pm on Mondays A couple of exceptions in the semester due to guest lecturer availability A couple of exceptions in the semester due to guest lecturer availability Coordinator: Kalina Yacef Coordinator: Kalina Yacef Lectures given by the coordinator and invited lecturers (IT academics, learning centre staff, librarians) Lectures given by the coordinator and invited lecturers (IT academics, learning centre staff, librarians) No textbook, online resources – check the web page No textbook, online resources – check the web page Assignments Assignments 1 – search results, w5 1 – search results, w5 2 - literature review and outline of research (25%), w7 2 - literature review and outline of research (25%), w7 3 - presentation (15%) +feedback on other presentations (10%) w10-13 3 - presentation (15%) +feedback on other presentations (10%) w10-13 seminar attendance required – max penalty 5% seminar attendance required – max penalty 5%

4. 4INFO5993 Research Methods, s1 2009 Topics Overview Introduction to research – definition, components, process, how to find a research question Introduction to research – definition, components, process, how to find a research question Types of research publications, quality metrics Types of research publications, quality metrics Literature review, how to search for relevant publications Literature review, how to search for relevant publications Writing a literature review and research proposal Writing a literature review and research proposal Oral presentation skills Oral presentation skills Research methods in IT (statistical analysis, mathematical analysis, algorithm analysis, simulation, qualitative analysis, etc.) Research methods in IT (statistical analysis, mathematical analysis, algorithm analysis, simulation, qualitative analysis, etc.) Ethics. Avoiding plagiarism. Intellectual Property. Ethics. Avoiding plagiarism. Intellectual Property.

5. 5INFO5993 Research Methods, s1 2009 Definition of Research 1) From the Merriam-Webster dictionary: 1: careful or diligent search 1: careful or diligent search 2: studious inquiry or examination; especially: investigation or experimentation aimed at the discovery and interpretation of facts, revision of accepted theories or laws in the light of new facts, or practical application of such new or revised theories or laws 2: studious inquiry or examination; especially: investigation or experimentation aimed at the discovery and interpretation of facts, revision of accepted theories or laws in the light of new facts, or practical application of such new or revised theories or laws 3: the collecting of information about a particular subject 3: the collecting of information about a particular subject 2) Booth, Columb & Williams, “The Craft of Research”: “Research is gathering information that answers a question and so solves a problem.” “Research is gathering information that answers a question and so solves a problem.”

6. 6INFO5993 Research Methods, s1 2009 Is This Research? To understand political decisions, a journalist finds out who contributed to election campaign fund To understand political decisions, a journalist finds out who contributed to election campaign fund To buy a laptop, a student compares various brands, configurations and prices To buy a laptop, a student compares various brands, configurations and prices To help companies stay competitive, a market researcher collects and interprets information To help companies stay competitive, a market researcher collects and interprets information To fix a computer, a technician finds out what procedure to use To fix a computer, a technician finds out what procedure to use

7. 7INFO5993 Research Methods, s1 2009 Academic Research In academic research, you must not only answer a question, but you must find something new and interesting In academic research, you must not only answer a question, but you must find something new and interesting You join a community of researchers You join a community of researchers You must advance the collective understanding of this community You must advance the collective understanding of this community Each community has a cumulative tradition with a set of interesting questions, tools and methods, practices, a style and language for writing up the research Each community has a cumulative tradition with a set of interesting questions, tools and methods, practices, a style and language for writing up the research Research is a conversation and ongoing social activity! Research is a conversation and ongoing social activity! You need critical and careful reading of published research You need critical and careful reading of published research to learn what the community already knows to learn what the community already knows to fit your work into the community to fit your work into the community to be prepared for your own work to be evaluated to be prepared for your own work to be evaluated

8. 8INFO5993 Research Methods, s1 2009 Key Components of Research A question of interest (research question) A question of interest (research question) A claim (contribution) A claim (contribution) Evidence Evidence Argument (links evidence to claim) Argument (links evidence to claim)

9. 9INFO5993 Research Methods, s1 2009 A Research Question Every piece of research should address a question of interest to the community Every piece of research should address a question of interest to the community Each community has traditional questions: Each community has traditional questions: What happens? Why does it happen? How should one do something? What something should one do? What happens? Why does it happen? How should one do something? What something should one do? Many questions fit into an on-going agenda, e.g. Many questions fit into an on-going agenda, e.g. Data mining foundations – mining sequential data; high- performance implementations of data mining algorithms, etc. Data mining foundations – mining sequential data; high- performance implementations of data mining algorithms, etc. Mining emerging data - e-commerce, web search data, moving object data, data from sensor networks Mining emerging data - e-commerce, web search data, moving object data, data from sensor networks … See a recent Conference Call for Papers See a recent Conference Call for Papers

10. 10INFO5993 Research Methods, s1 2009 A Claim (Contribution) Every piece of research makes a claim (the “contribution”) answering a research question Every piece of research makes a claim (the “contribution”) answering a research question Claims can be very diverse among fields and within fields Claims can be very diverse among fields and within fields Ex. for a “what happens” question - when using weak concurrency control, how often is the data corrupted Ex. for a “what happens” question - when using weak concurrency control, how often is the data corrupted Ex. for a “ why something happens” - what factors lead to project success in open-source development Ex. for a “ why something happens” - what factors lead to project success in open-source development Ex. for a “better way to do something” - modifying algorithm X in a particular way improves its performance (speed, accuracy, etc) Ex. for a “better way to do something” - modifying algorithm X in a particular way improves its performance (speed, accuracy, etc) Ex. for a “better something to do” - our system allows users to see the model of their skills kept in a teaching system Ex. for a “better something to do” - our system allows users to see the model of their skills kept in a teaching system Be explicit about the meaning of “better”

11. 11INFO5993 Research Methods, s1 2009 Evidence You must back up the claim with evidence, e.g. You must back up the claim with evidence, e.g. Empirical evaluation of a machine learning algorithm to evaluate its accuracy Empirical evaluation of a machine learning algorithm to evaluate its accuracy Analysis of the computational complexity of an algorithm Analysis of the computational complexity of an algorithm A mathematical proof to show that some process/algorithm has desired properties A mathematical proof to show that some process/algorithm has desired properties A prototype implementation to show that a system can be built to achieve the claimed functionality A prototype implementation to show that a system can be built to achieve the claimed functionality A simulation model which is executed and analysed to show certain properties A simulation model which is executed and analysed to show certain properties Measurements of a running system to show it has good performance Measurements of a running system to show it has good performance Observations of behaviour in an organisation to show what is happening Observations of behaviour in an organisation to show what is happening Various research methods, each defined by the sort of evidence that it can produce Various research methods, each defined by the sort of evidence that it can produce each community has its own standards of quality and reasonableness each community has its own standards of quality and reasonableness

12. 12INFO5993 Research Methods, s1 2009 Argument You should show that the evidence you offer supports the claim you make You should show that the evidence you offer supports the claim you make It is essential that you deal with natural or obvious objections to the correctness or importance of the work It is essential that you deal with natural or obvious objections to the correctness or importance of the work that is, you must think like your readers, and anticipate their reactions that is, you must think like your readers, and anticipate their reactions In systems work, this is often called an “evaluation” of the design In systems work, this is often called an “evaluation” of the design

13. 13INFO5993 Research Methods, s1 2009 Research Paper - Example Identify the Identify the Research question Research question Claim Claim Evidence Evidence Argument Argument

14. 14INFO5993 Research Methods, s1 2009 Claim and Argument - Examples This system design leads to better performance on some metric This system design leads to better performance on some metric make sure you limit how much worse this makes other metrics (such as cost!) make sure you limit how much worse this makes other metrics (such as cost!) make sure your measurements are fair (don’t compare with “strawman” design but with state-of-the-art) make sure your measurements are fair (don’t compare with “strawman” design but with state-of-the-art) This system design offers better functionality for some uses This system design offers better functionality for some uses make sure you show it can be implemented with adequate performance make sure you show it can be implemented with adequate performance

15. 15INFO5993 Research Methods, s1 2009 Claim and Argument – Examples (2) This behaviour can be explained by this theory This behaviour can be explained by this theory make sure you don’t have confounding factors such as level of experience, or method novelty, or subject expectations (“placebo effect”) make sure you don’t have confounding factors such as level of experience, or method novelty, or subject expectations (“placebo effect”) This is what happens This is what happens make sure you don’t interfere too much with what happens when you gather data, or misinterpret it due to observer expectations make sure you don’t interfere too much with what happens when you gather data, or misinterpret it due to observer expectations

16. 16INFO5993 Research Methods, s1 2009 Common Mistakes 1 Gather lots of data without a focussed question or method Gather lots of data without a focussed question or method A collection of facts is not a contribution! A collection of facts is not a contribution! it must reveal some pattern or understanding that you make explicit it must reveal some pattern or understanding that you make explicit

17. 17INFO5993 Research Methods, s1 2009 Common Mistakes 2 Build a system without a focused question or planned evaluation Build a system without a focused question or planned evaluation E.g. let’s see how to use aspect-oriented programming in a sensor network E.g. let’s see how to use aspect-oriented programming in a sensor network An innovative system is not a contribution! An innovative system is not a contribution! it must be a worthwhile innovation in a sense you make explicit it must be a worthwhile innovation in a sense you make explicit E.g. better performance E.g. better performance E.g. new functionality E.g. new functionality

18. 18INFO5993 Research Methods, s1 2009 Negative Results Sometimes, you don’t get the result you hoped for Sometimes, you don’t get the result you hoped for You gather data that does not reveal any pattern or understanding You gather data that does not reveal any pattern or understanding E.g. no factor seems to correlate well with project success E.g. no factor seems to correlate well with project success You design a system that turns out to be worse than the state-of-the-art You design a system that turns out to be worse than the state-of-the-art E.g. your machine learning algorithm runs slower than expected E.g. your machine learning algorithm runs slower than expected You can still salvage a thesis You can still salvage a thesis Try to find some way to contribute to our understanding, or suggest fruitful directions for further work Try to find some way to contribute to our understanding, or suggest fruitful directions for further work E.g. what features of the algorithm make it slow E.g. what features of the algorithm make it slow Make sure the problem is intrinsic, not just your bad coding/experiment design/etc Make sure the problem is intrinsic, not just your bad coding/experiment design/etc

19. 19INFO5993 Research Methods, s1 2009 Ground-Breaking Work Very rarely, a piece of research will establish a whole new agenda for a field, or even a new field Very rarely, a piece of research will establish a whole new agenda for a field, or even a new field the contribution can be as much in the possibilities for further work, as in the result itself! the contribution can be as much in the possibilities for further work, as in the result itself! In some sense, this is work that asks a new type of question, or introduces a new method In some sense, this is work that asks a new type of question, or introduces a new method We don’t recommend this for Hons/MIT/MSc/PhD We don’t recommend this for Hons/MIT/MSc/PhD save the idea till you have time enough, and flexibility enough to deal with inevitable digressions/difficulties save the idea till you have time enough, and flexibility enough to deal with inevitable digressions/difficulties

20. INFO5993 Research Methods, s1 2009 20 Great scholars do not solve problems; they create them. -Albert Einstein

21. 21INFO5993 Research Methods, s1 2009 Idealised Research Process Find a question to seek an answer for Find a question to seek an answer for Method: Choose an appropriate research method and make flexible plans Method: Choose an appropriate research method and make flexible plans Evidence: Gather the data, do the experiment, build the prototype etc. Evidence: Gather the data, do the experiment, build the prototype etc. Contribution: Analyse, interpret, and conclude Contribution: Analyse, interpret, and conclude Argument: Write the report Argument: Write the report Importance of “writing” (aided by thinking from the point of view of your readers) Importance of “writing” (aided by thinking from the point of view of your readers)

22. 22INFO5993 Research Methods, s1 2009 Actual Research Process Research explores new areas and the results are not predictable! Research explores new areas and the results are not predictable! The research plan is iterative The research plan is iterative Gathering evidence leads to changes to the claim Gathering evidence leads to changes to the claim sometimes one refines the claim sometimes one refines the claim E.g. limit the scope E.g. limit the scope from “algorithm X outperforms Y” to “algorithm X outperforms Y when the independence assumption is violated” from “algorithm X outperforms Y” to “algorithm X outperforms Y when the independence assumption is violated” From “X has higher throughput” to “X has higher throughput if the contention rate is low” From “X has higher throughput” to “X has higher throughput if the contention rate is low” sometimes one must change the claim entirely sometimes one must change the claim entirely sometimes while gathering evidence, one finds new questions which look worth answering! sometimes while gathering evidence, one finds new questions which look worth answering! New claims or questions need further evidence, revised plans, maybe even different methods New claims or questions need further evidence, revised plans, maybe even different methods

23. 23INFO5993 Research Methods, s1 2009 The “Great Expedition into Unknown Terrain” metaphor Imke Tammen Imke Tammen http://www.itl.usyd.edu.au/supervision/casestudies/ casestudy.cfm?id=8 students and supervisors as co-explorers students and supervisors as co-explorers

24. 24INFO5993 Research Methods, s1 2009 Finding a Question Especially when you are learning to do research, it may be already chosen for you by supervisor Especially when you are learning to do research, it may be already chosen for you by supervisor or supervisor may suggest an area, and leave you to find the question or supervisor may suggest an area, and leave you to find the question A question may arise from some previous research A question may arise from some previous research Further work, issues not addressed, holes in the evidence collected Further work, issues not addressed, holes in the evidence collected A question may come from the combination of previous research A question may come from the combination of previous research Bring two areas together, use a technique from one area in another Bring two areas together, use a technique from one area in another A question may arise due to new technology A question may arise due to new technology new hardware or technique may require new models, new hardware may influence use or performance or feasibility new hardware or technique may require new models, new hardware may influence use or performance or feasibility

25. 25INFO5993 Research Methods, s1 2009 Suitable Research Questions Answerability – can the questions be answered through research? Answerability – can the questions be answered through research? Scale: Consider available resources (equipment, time, skills) Scale: Consider available resources (equipment, time, skills) Scope: Often start with broad topic space/ bigger question, then narrow in to a specific question Scope: Often start with broad topic space/ bigger question, then narrow in to a specific question

26. 26INFO5993 Research Methods, s1 2009 Tips for Finding Research Questions Try the research topic generator http://www.cs.purdue.edu/homes/dec/essay.topic.generator.html Try the research topic generator http://www.cs.purdue.edu/homes/dec/essay.topic.generator.html

27. 27INFO5993 Research Methods, s1 2009 Tips for Finding Research Questions (2) Read the papers your supervisor gave you Read the papers your supervisor gave you follow the references, check the web pages of the authors follow the references, check the web pages of the authors read carefully the “Future research” sections read carefully the “Future research” sections write down your ideas!! write down your ideas!! Find the top conferences in your field Find the top conferences in your field scan the call for papers and associated workshops for hot topics scan the call for papers and associated workshops for hot topics scan the conference proceedings to identify important topics, key people and research groups. Check their web pages. scan the conference proceedings to identify important topics, key people and research groups. Check their web pages. Find review (survey) articles Find review (survey) articles

28. 28INFO5993 Research Methods, s1 2009 Tips for Finding Research Questions (2) Callahan, 2001

29. 29INFO5993 Research Methods, s1 2009 Describing Your Research Problem You need several clear, concise and succinct statements of the research problem of different lengths You need several clear, concise and succinct statements of the research problem of different lengths e.g. one minute (elevator) pitch e.g. one minute (elevator) pitch e.g. ten minutes introduction to full seminar e.g. ten minutes introduction to full seminar Issues you must deal with: Issues you must deal with: Can it be understood by others without too much background? Can it be understood by others without too much background? Does it demonstrate a good understanding of the research community? Does it demonstrate a good understanding of the research community?

30. 30INFO5993 Research Methods, s1 2009 Guide to Research Literature Types of publications Types of publications conference and workshop papers conference and workshop papers journal papers journal papers technical reports technical reports monographs monographs

31. 31INFO5993 Research Methods, s1 2009 Conference Papers Call for papers - ~1 year before meeting Call for papers - ~1 year before meeting Paper submission - ~4-8 months before meeting Paper submission - ~4-8 months before meeting Page limit e.g. 8 pages Page limit e.g. 8 pages Details often omitted (proofs, design technicalities) Details often omitted (proofs, design technicalities) Program Committee reviews the papers Program Committee reviews the papers Criteria: significance, originality, soundness, readability Criteria: significance, originality, soundness, readability Final version for proceedings due ~3 months before meeting Final version for proceedings due ~3 months before meeting revise by author in light of reviews revise by author in light of reviews but not checked again but not checked again Annual or bi-annual conferences Annual or bi-annual conferences

32. 32INFO5993 Research Methods, s1 2009 Selection Process Typically 3 reviewers Typically 3 reviewers Acceptance rate – varies Acceptance rate – varies Some 10-15%, others 50% Some 10-15%, others 50% Some review “blind” (author details not shown to reviewers), others do not Some review “blind” (author details not shown to reviewers), others do not - Example – a reviewer’s form - Ask your supervisor for guidance about which are the reliable and important conferences in your field!

33. 33INFO5993 Research Methods, s1 2009 Standard of conferences Identify the top conferences in your area Identify the top conferences in your area Ask your supervisor Ask your supervisor CORE ranking (sometimes inaccurate) CORE ranking (sometimes inaccurate) Affiliation (ACM, IEEE, etc..) Affiliation (ACM, IEEE, etc..) Acceptance rate and review process Acceptance rate and review process

34. 34INFO5993 Research Methods, s1 2009 “I regret to inform you…” When a submission is not accepted by a conference When a submission is not accepted by a conference The author should use the reviewer’s comments to revise and improve the paper, e.g. The author should use the reviewer’s comments to revise and improve the paper, e.g. if reviewer misunderstood something, author explains it more clearly if reviewer misunderstood something, author explains it more clearly if reviewer points to missing citations, author adds them if reviewer points to missing citations, author adds them If reviewer is not convinced, author can do more experiments If reviewer is not convinced, author can do more experiments Then submit revised paper to another conference in the same community Then submit revised paper to another conference in the same community Often the resubmission is to a lower prestige conference Often the resubmission is to a lower prestige conference Submit to the same conference next year? Not often – IT changes rapidly Submit to the same conference next year? Not often – IT changes rapidly

35. 35INFO5993 Research Methods, s1 2009 Workshop Papers A workshop is typically a smaller meeting than a conference A workshop is typically a smaller meeting than a conference Sometimes workshop papers are just like conference papers Sometimes workshop papers are just like conference papers Other workshops are more preliminary Other workshops are more preliminary can publish a position paper (draft of an idea without evidence, or proposal for future work) can publish a position paper (draft of an idea without evidence, or proposal for future work) less rigorously reviewed, the goal is mainly to allow the community to meet less rigorously reviewed, the goal is mainly to allow the community to meet

36. 36INFO5993 Research Methods, s1 2009 Journal Article Typically longer than a conference paper Typically longer than a conference paper Often based on a conference paper with additions, corrections and improvements Often based on a conference paper with additions, corrections and improvements Refereed by Refereed by at least 3 reviewers, experts in the field at least 3 reviewers, experts in the field they spend months on the paper checking details, etc. they spend months on the paper checking details, etc. Decisions: accepted, accepted with minor revisions, major revisions and resubmission, rejected Decisions: accepted, accepted with minor revisions, major revisions and resubmission, rejected Revisions, refereed again Revisions, refereed again Accepted, published after several months (journal issues have limited capacity) Accepted, published after several months (journal issues have limited capacity) Time from submission to publication varies, typically 1-1.5 years but may be 3-4 years Time from submission to publication varies, typically 1-1.5 years but may be 3-4 years

37. 37INFO5993 Research Methods, s1 2009 Standard of Journals Many journals in each area with different standards Many journals in each area with different standards Typically IEEE Transactions and ACM Communications are some of the top-ranked journals Typically IEEE Transactions and ACM Communications are some of the top-ranked journals Not all IEEE Trans. and ACM Comm. are top journals Not all IEEE Trans. and ACM Comm. are top journals Check CORE ranking (not always accurate though- check with your supervisor) Check CORE ranking (not always accurate though- check with your supervisor) Ask you supervisor which journals are the top-ranked and most important in your area!

38. 38INFO5993 Research Methods, s1 2009 Technical Report Issued by the author’s department, with a number and date Issued by the author’s department, with a number and date May be based on a conference paper May be based on a conference paper Longer, includes all the boring details that are omitted from the conference paper due to space limitations Longer, includes all the boring details that are omitted from the conference paper due to space limitations Used to establish priority Used to establish priority E.g. produce TR before submitting to conference or journal + conference and journal papers may get rejected E.g. produce TR before submitting to conference or journal + conference and journal papers may get rejected - Find the School of IT’s TRs!

39. 39INFO5993 Research Methods, s1 2009 PhD or MSc Thesis Very extensive account Very extensive account Show much of the research process Show much of the research process Extensive survey of the literature Extensive survey of the literature Very complete evaluation of the work Very complete evaluation of the work The goal is to establish that the author is ready to become independent researcher The goal is to establish that the author is ready to become independent researcher i.e. PhD and MSc provide research training i.e. PhD and MSc provide research training Typically checked by 2 or 3 reviewers Typically checked by 2 or 3 reviewers

40. 40INFO5993 Research Methods, s1 2009 Monograph A collection of selected papers from a conference or workshop A collection of selected papers from a conference or workshop A bit more checking than for the conference/workshop A bit more checking than for the conference/workshop An author can offer a coherent and unified account of a whole research topic An author can offer a coherent and unified account of a whole research topic often combines their own results with other people’s often combines their own results with other people’s Revisits several papers using unified notation, better exposition, better literature review, etc. Revisits several papers using unified notation, better exposition, better literature review, etc. Publisher may get reviewers but their focus is “will it sell” not “is it correct” Publisher may get reviewers but their focus is “will it sell” not “is it correct”

41. 41INFO5993 Research Methods, s1 2009 Warnings Quality of conferences and journals varies, and this is reflected in the checking of the papers Quality of conferences and journals varies, and this is reflected in the checking of the papers Read papers with a critical eye! Read papers with a critical eye! Some communities are very clique-dominated Some communities are very clique-dominated Unpopular opinions are not welcome Unpopular opinions are not welcome Clique leaders can publish anything, even half-baked ideas without evidence Clique leaders can publish anything, even half-baked ideas without evidence

42. 42INFO5993 Research Methods, s1 2009 Fake Conferences and Random Papers http://pdos.csail.mit.edu/scigen/ http://pdos.csail.mit.edu/scigen/ http://pdos.csail.mit.edu/scigen/ A random paper accepted to a journal? A random paper accepted to a journal?

43. 43INFO5993 Research Methods, s1 2009 The Research Community A community has conferences and journals of high prestige which they read and publish in A community has conferences and journals of high prestige which they read and publish in They meet often, and each knows (more or less) what others are doing They meet often, and each knows (more or less) what others are doing You must place your work in the context of a community You must place your work in the context of a community Divided geographically Divided geographically Europe vs America vs Asia Europe vs America vs Asia

44. 44INFO5993 Research Methods, s1 2009 Quality Metrics How important is an article? How influential is an author? How important is an article? How influential is an author? Based on citation analysis - number of times a paper or author is cited Based on citation analysis - number of times a paper or author is cited How to calculate citations – Google Scholar + other software How to calculate citations – Google Scholar + other software Assumption: important authors and articles are cited more often than the others Assumption: important authors and articles are cited more often than the others Increasingly used by governments, funding bodies, promotion committees to evaluate the quality of author’s work Increasingly used by governments, funding bodies, promotion committees to evaluate the quality of author’s work Some drawbacks Some drawbacks Citing errors – authors with the same names are not separated Citing errors – authors with the same names are not separated Cliques (friends, colleagues) cite each other in turn to build their citation index Cliques (friends, colleagues) cite each other in turn to build their citation index Negative citations are included (citations to incorrect results) Negative citations are included (citations to incorrect results)

45. 45INFO5993 Research Methods, s1 2009 ISI Citation Database Very popular, established in 1960, contains >40million records, contains Very popular, established in 1960, contains >40million records, contains Arts and Humanities Citation Index (A&HCI) Arts and Humanities Citation Index (A&HCI) Science Citation Index (SCI) Science Citation Index (SCI) Social Sciences Citation Index (SSCI) Social Sciences Citation Index (SSCI) However However it doesn’t index a large number of journals it doesn’t index a large number of journals ignores open-access journals ignores open-access journals doesn’t index conferences doesn’t index conferences “Read the Rise and Rise of Citation Analysis” by L. Meho!

46. 46INFO5993 Research Methods, s1 2009 Journal’s Impact Factor Journal impact factors Journal impact factors Used to determine the importance of a journal Used to determine the importance of a journal E.g. journal impact factor for 2007 = E.g. journal impact factor for 2007 = # citations in 2007 to articles published in the journal in 2005-6 # citations in 2007 to articles published in the journal in 2005-6 = ------------------------------------------------------------------------------- # articles published in the journal in 2005-6 # articles published in the journal in 2005-6 Check CS journal impact factors on ISI Web of Knowledge! (electronic journals -> J->Journal Citation Report)

47. 47INFO5993 Research Methods, s1 2009 CORE’s ratings Computing Research and Education Association of Australasia (CORE) Computing Research and Education Association of Australasia (CORE) Australia and New Zealand Australia and New Zealand Ranking of journals and conferences in CS – not finalised Ranking of journals and conferences in CS – not finalisedhttp://www.core.edu.au/

48. 48INFO5993 Research Methods, s1 2009 Author’s Citation Indexes for Measuring Impact total number of citations total number of citations h-index h-index proposed by J.E. Hirsh in 2005: proposed by J.E. Hirsh in 2005: “A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np-h) papers have no more than h citations each. “ What is the h-index? What is the h-index? Paper 1: 20 citations Paper 2: 15 citations Paper 3: 8 citations Paper 4: 4 citations Paper 5: 3 citations … An h-index of 4 means that there are at least 4 papers cited at least 4 times each.

49. 49INFO5993 Research Methods, s1 2009 Author’s Citation Indexes for Measuring Impact (2) g-index g-index Proposed by L. Egghe 2006: Proposed by L. Egghe 2006: “Given a set of articles ranked in decreasing order of the number of citations that they received, the g-index is the (unique) largest number such that the top g articles received (together) at least g*g citations.” improves h-index by giving more weight to highly cited articles improves h-index by giving more weight to highly cited articles Several variants of h-index and g-index Several variants of h-index and g-index Calculate the g-index for the example from the previous slide!

50. 50INFO5993 Research Methods, s1 2009 h vs g index Rank # cit. 15 25 35 45 55 65 75 85 95 105Rank 1100 250 350 450 55 65 75 85 95 105 R2R2R2R2∑cit15 410 915 1620 2525 3630 4935 6440 8145 10050 R2R2R2R2∑cit1100 4150 9200 16250 25255 36260 49265 64270 81275 100280 h-index? A B g index?

51. 51INFO5993 Research Methods, s1 2009 h vs g index Rank # cit. 115 125 135 145 155 165 175 185 195 205 R2R2R2R2∑cit121285 144295 169300 196305 225310 256315 289320 324325 361330 400335

52. 52INFO5993 Research Methods, s1 2009 Publish or Perish http://www.harzing.com/resources.htm#/pop.htm http://www.harzing.com/resources.htm#/pop.htm http://www.harzing.com/resources.htm#/pop.htm Perform a citation analysis of your supervisor’s publications! What are the limitations of Perform a citation analysis of your supervisor’s publications! What are the limitations of citation analysis in general? citation analysis in general? g- and h-indexes as citation metrics? g- and h-indexes as citation metrics?