1. 1INFO4990 Research Methods, s2 2008 INFO4990 Research Methods Bing Bing Zhou http://www.cs.usyd.edu.au/~info4990/ Lecture based in part on materials by Alan Fekete, Mary Lou Maher, Joseph Davis, Irena Koprinska and others Research – Components and Process

2. 2INFO4990 Research Methods, s2 2008 Outline Administrative matters Administrative matters Research Research Definition, key components, process Definition, key components, process Finding a research question Finding a research question

3. 3INFO4990 Research Methods, s2 2008 Administrativia Course web page: http://www.cs.usyd.edu.au/~info4990/ Course web page: http://www.cs.usyd.edu.au/~info4990/ 2 hours lectures/workshops, 4-6pm on Mondays 2 hours lectures/workshops, 4-6pm on Mondays Coordinator: Bing Bing Zhou Coordinator: Bing Bing Zhou Lectures given by the coordinator and invited lecturers (IT academics, learning centre staff, librarians) Lectures given by the coordinator and invited lecturers (IT academics, learning centre staff, librarians) Consultation time: Monday 3-4pm Consultation time: Monday 3-4pm

4. 4INFO4990 Research Methods, s2 2008 Administrativia No textbook, on-line resources – check the web page No textbook, on-line resources – check the web page Assignments Assignments 1 – search results (15%), w4 1 – search results (15%), w4 2 - literature review and outline of research (45%), w8 2 - literature review and outline of research (45%), w8 3 - presentation (30%) +feedback on other presentations (10%) w12-13 3 - presentation (30%) +feedback on other presentations (10%) w12-13 Basser seminar attendance required – max penalty 5% Basser seminar attendance required – max penalty 5%

5. 5INFO4990 Research Methods, s2 2008 Topics Overview Introduction to research – definition, components, process, how to find a research question Introduction to research – definition, components, process, how to find a research question Literature review Literature review how to search for relevant publications how to search for relevant publications Writing a literature review and research proposal Writing a literature review and research proposal Oral presentation skills Oral presentation skills Research methods in IT (statistical analysis, mathematical analysis, algorithm analysis, simulation, qualitative analysis, ethics, etc.) Research methods in IT (statistical analysis, mathematical analysis, algorithm analysis, simulation, qualitative analysis, ethics, etc.)

6. 6INFO4990 Research Methods, s2 2008 Definition of Research 1) From the Merriam-Webster dictionary: 1: careful or diligent search 1: careful or diligent search 2: studious inquiry or examination; especially: investigation or experimentation aimed at the discovery and interpretation of facts, revision of accepted theories or laws in the light of new facts, or practical application of such new or revised theories or laws 2: studious inquiry or examination; especially: investigation or experimentation aimed at the discovery and interpretation of facts, revision of accepted theories or laws in the light of new facts, or practical application of such new or revised theories or laws 3: the collecting of information about a particular subject 3: the collecting of information about a particular subject 2) Booth, Columb & Williams, “The Craft of Research”: “Research is gathering information that answers a question and so solves a problem.” “Research is gathering information that answers a question and so solves a problem.”

7. 7INFO4990 Research Methods, s2 2008 Is This Research? To understand political decisions, a journalist finds out who contributed to election campaign fund To understand political decisions, a journalist finds out who contributed to election campaign fund To buy a laptop, a student compares various brands, configurations and prices To buy a laptop, a student compares various brands, configurations and prices To help companies stay competitive, a market researcher collects and interprets information To help companies stay competitive, a market researcher collects and interprets information To fix a computer, a technician finds out what procedure to use To fix a computer, a technician finds out what procedure to use

8. 8INFO4990 Research Methods, s2 2008 Academic Research In academic research, you must not only answer a question, In academic research, you must not only answer a question, but you must find something new and interesting but you must find something new and interestingand you must advance the collective understanding of the community you must advance the collective understanding of the community Each community has a cumulative tradition with a set of interesting questions, tools and methods, practices, a style and language for writing up the research Each community has a cumulative tradition with a set of interesting questions, tools and methods, practices, a style and language for writing up the research Research is also a conversation and ongoing social activity! Research is also a conversation and ongoing social activity! You need critical and careful reading of published research You need critical and careful reading of published research to learn what the community already knows to learn what the community already knows to fit your work into the community to fit your work into the community to be prepared for your own work to be evaluated to be prepared for your own work to be evaluated

9. 9INFO4990 Research Methods, s2 2008 Key Components of Research A question of interest (research question) A question of interest (research question) A claim (contribution) A claim (contribution) Evidence Evidence Argument (links evidence to claim) Argument (links evidence to claim)

10. 10INFO4990 Research Methods, s2 2008 A Research Question Every piece of research should address a question of interest to the community Every piece of research should address a question of interest to the community Each community has traditional questions: Each community has traditional questions: What happens? Why does it happen? How should one do something? What something should one do? What happens? Why does it happen? How should one do something? What something should one do? Many questions fit into an on-going agenda, e.g. Many questions fit into an on-going agenda, e.g. Data mining foundations – mining sequential data; high- performance implementations of data mining algorithms, etc. Data mining foundations – mining sequential data; high- performance implementations of data mining algorithms, etc. Mining emerging data - e-commerce, web search data, moving object data, data from sensor networks Mining emerging data - e-commerce, web search data, moving object data, data from sensor networks … See a Conference Call for Papers See a Conference Call for Papers

11. 11INFO4990 Research Methods, s2 2008 A Claim (Contribution) Every piece of research makes a claim (the “contribution”) answering a research question Every piece of research makes a claim (the “contribution”) answering a research question Claims can be very diverse among fields and within fields Claims can be very diverse among fields and within fields

12. 12INFO4990 Research Methods, s2 2008 Evidence You must back up the claim with evidence, e.g. You must back up the claim with evidence, e.g. A mathematical proof to show that some process/algorithm has desired properties A mathematical proof to show that some process/algorithm has desired properties Analysis of the computational complexity of an algorithm Analysis of the computational complexity of an algorithm Empirical evaluation of a machine learning algorithm to evaluate its accuracy Empirical evaluation of a machine learning algorithm to evaluate its accuracy A simulation model which is executed and analysed to show certain properties A simulation model which is executed and analysed to show certain properties A prototype implementation to show that a system can be built to achieve the claimed functionality A prototype implementation to show that a system can be built to achieve the claimed functionality Measurements of a running system to show it has good performance Measurements of a running system to show it has good performance Observations of behaviour in an organisation to show what is happening Observations of behaviour in an organisation to show what is happening Various research methods, each defined by the sort of evidence that it can produce Various research methods, each defined by the sort of evidence that it can produce each community has its own standards of quality and reasonableness each community has its own standards of quality and reasonableness

13. 13INFO4990 Research Methods, s2 2008 Argument You should show that the evidence you offer supports the claim you make You should show that the evidence you offer supports the claim you make It’s essential that you deal with natural or obvious objections to the correctness or importance of the work It’s essential that you deal with natural or obvious objections to the correctness or importance of the work that is, you must think like your readers, and anticipate their reactions that is, you must think like your readers, and anticipate their reactions In systems work, this is often called an “evaluation” of the design In systems work, this is often called an “evaluation” of the design

14. 14INFO4990 Research Methods, s2 2008 Claim and Argument - Examples This system design leads to better performance on some metric This system design leads to better performance on some metric make sure you limit how much worse this makes other metrics (such as cost!) make sure you limit how much worse this makes other metrics (such as cost!) make sure your measurements are fair (don’t compare with “strawman” design but with state-of-the-art) make sure your measurements are fair (don’t compare with “strawman” design but with state-of-the-art) This system design offers better functionality for some uses This system design offers better functionality for some uses make sure you show it can be implemented with adequate performance make sure you show it can be implemented with adequate performance

15. 15INFO4990 Research Methods, s2 2008 Claim and Argument – Examples (2) This behaviour can be explained by this theory This behaviour can be explained by this theory make sure you don’t have confounding factors such as level of experience, or method novelty, or subject expectations make sure you don’t have confounding factors such as level of experience, or method novelty, or subject expectations This is what happens This is what happens make sure you don’t interfere too much with what happens when you gather data, or misinterpret it due to observer expectations make sure you don’t interfere too much with what happens when you gather data, or misinterpret it due to observer expectations

16. 16INFO4990 Research Methods, s2 2008 Common Mistakes 1 Gather lots of data without a focussed question or method Gather lots of data without a focussed question or method A collection of facts is not a contribution! A collection of facts is not a contribution! it must reveal some pattern or understanding that you make explicit it must reveal some pattern or understanding that you make explicit

17. 17INFO4990 Research Methods, s2 2008 Common Mistakes 2 Build a system without a focused question or planned evaluation Build a system without a focused question or planned evaluation E.g. let’s see how to use aspect-oriented programming in a sensor network E.g. let’s see how to use aspect-oriented programming in a sensor network An innovative system is not a contribution! An innovative system is not a contribution! it must be a worthwhile innovation in a sense you make explicit it must be a worthwhile innovation in a sense you make explicit E.g. better performance E.g. better performance E.g. new functionality E.g. new functionality

18. 18INFO4990 Research Methods, s2 2008 Negative Results Sometimes, you don’t get the result you hoped for Sometimes, you don’t get the result you hoped for You gather data that does not reveal any pattern or understanding You gather data that does not reveal any pattern or understanding E.g. no factor seems to correlate well with project success E.g. no factor seems to correlate well with project success You design a system that turns out to be worse than the state-of-the-art You design a system that turns out to be worse than the state-of-the-art E.g. your machine learning algorithm runs slower than expected E.g. your machine learning algorithm runs slower than expected You can still salvage a thesis You can still salvage a thesis Try to find some way to contribute to our understanding, or suggest fruitful directions for further work Try to find some way to contribute to our understanding, or suggest fruitful directions for further work E.g. what features of the algorithm make it slow E.g. what features of the algorithm make it slow Make sure the problem is intrinsic, not just your bad coding/experiment design/etc Make sure the problem is intrinsic, not just your bad coding/experiment design/etc

19. 19INFO4990 Research Methods, s2 2008 Ground-Breaking Work Very rarely, a piece of research will establish a whole new agenda for a field, or even a new field Very rarely, a piece of research will establish a whole new agenda for a field, or even a new field the contribution can be as much in the possibilities for further work, as in the result itself! the contribution can be as much in the possibilities for further work, as in the result itself! In some sense, this is work that asks a new type of question, or introduces a new method In some sense, this is work that asks a new type of question, or introduces a new method We don’t recommend this for Hons/MIT/MSc/MEng We don’t recommend this for Hons/MIT/MSc/MEng save the idea till you have time enough, and flexibility enough to deal with inevitable digressions/difficulties save the idea till you have time enough, and flexibility enough to deal with inevitable digressions/difficulties

20. INFO4990 Research Methods, s2 2008 20 Great scholars do not solve problems; they create them. -Albert Einstein

21. 21INFO4990 Research Methods, s2 2008 Idealised Research Process Find a question to seek an answer for Find a question to seek an answer for Method: Choose an appropriate research method and make flexible plans Method: Choose an appropriate research method and make flexible plans Evidence: Gather the data, do the experiment, build the prototype etc. Evidence: Gather the data, do the experiment, build the prototype etc. Contribution: Analyse, interpret, and conclude Contribution: Analyse, interpret, and conclude Argument: Write the report Argument: Write the report Importance of “writing” (aided by thinking from the point of view of your readers) Importance of “writing” (aided by thinking from the point of view of your readers)

22. 22INFO4990 Research Methods, s2 2008 Actual Research Process Research explores new areas and the results are not predictable! Research explores new areas and the results are not predictable! The research plan is iterative The research plan is iterative Gathering evidence leads to changes to the claim Gathering evidence leads to changes to the claim sometimes one refines the claim sometimes one refines the claim E.g. limit the scope E.g. limit the scope from “algorithm X outperforms Y” to “algorithm X outperforms Y when the independence assumption is violated” from “algorithm X outperforms Y” to “algorithm X outperforms Y when the independence assumption is violated” From “X has higher throughput” to “X has higher throughput if the contention rate is low” From “X has higher throughput” to “X has higher throughput if the contention rate is low” sometimes one must change the claim entirely sometimes one must change the claim entirely sometimes while gathering evidence, one finds new questions which look worth answering! sometimes while gathering evidence, one finds new questions which look worth answering! New claims or questions need further evidence, revised plans, maybe even different methods New claims or questions need further evidence, revised plans, maybe even different methods

23. 23INFO4990 Research Methods, s2 2008 Finding a Question Especially when you are learning to do research, it may be already chosen for you by supervisor Especially when you are learning to do research, it may be already chosen for you by supervisor or supervisor may suggest an area, and leave you to find the question or supervisor may suggest an area, and leave you to find the question A question may arise from some previous research A question may arise from some previous research Further work, issues not addressed, holes in the evidence collected Further work, issues not addressed, holes in the evidence collected A question may come from the combination of previous research A question may come from the combination of previous research Bring two areas together, use a technique from one area in another Bring two areas together, use a technique from one area in another A question may arise due to new technology A question may arise due to new technology new hardware or technique may require new models, new hardware may influence use or performance or feasibility new hardware or technique may require new models, new hardware may influence use or performance or feasibility

24. 24INFO4990 Research Methods, s2 2008 Suitable Research Questions Answerability – can the questions be answered through research? Answerability – can the questions be answered through research? Scale: Consider available resources (equipment, time, skills) Scale: Consider available resources (equipment, time, skills) Scope: Often start with broad topic space/ bigger question, then narrow in to a specific question Scope: Often start with broad topic space/ bigger question, then narrow in to a specific question

25. 25INFO4990 Research Methods, s2 2008 Tips for Finding Research Questions Read the papers you supervisor gave you Read the papers you supervisor gave you follow the references, check the web pages of the authors follow the references, check the web pages of the authors read carefully the “Future research” sections read carefully the “Future research” sections write down your ideas!! write down your ideas!! Find the top conferences/journals in your field Find the top conferences/journals in your field scan the call for papers and associated workshops for hot topics scan the call for papers and associated workshops for hot topics scan the conference proceedings/journals to identify important topics, key people and research groups. Check their web pages. scan the conference proceedings/journals to identify important topics, key people and research groups. Check their web pages. Find review (survey) articles Find review (survey) articles

26. 26INFO4990 Research Methods, s2 2008 Tips for Finding Research Questions Callahan, 2001

27. 27INFO4990 Research Methods, s2 2008 Describing Your Research Problem You need several clear, concise and succinct statements of the research problem of different lengths You need several clear, concise and succinct statements of the research problem of different lengths e.g. one minute (elevator) pitch e.g. one minute (elevator) pitch e.g. ten minutes introduction to full seminar e.g. ten minutes introduction to full seminar Issues you must deal with: Issues you must deal with: Can it be understood by others without too much background? Can it be understood by others without too much background? Does it demonstrate a good understanding of the research community? Does it demonstrate a good understanding of the research community?