Research Seminars in IT in Education (MIT6003) Research Methodology I Dr Jacky Pow

Agenda  Method evaluation: critical review of research procedures

Typical research procedures  Formulation of the research problem –An emerging issue  Review of the literature –What has been done and what need to be done  Data collection –How to collect the necessary data to answer the research question

Typical research procedures  Data analysis –How to analyze the and organize the data  Discussion –To discuss the implication of the data and see whether the research questions can be answered –Future research agenda  Conclusion –What this research has achieved

Class activity  What questions would you ask if you were to evaluate the appropriateness of the research method in a study?  How will you answer the questions you identified?

Issues in educational research  Reliability (Trochim, 2002) –In research context, the term reliability means repeatability or consistency. A measure (of the variables) is considered reliable if it would give us the same result over and over again (assuming that what we are measuring is not changing!)

Types of reliability  Inter-Rater/Inter-observer Reliability –Used to assess the degree to which different raters/observers give consistent estimates of the same phenomenon  Test-Retest reliability –Used to assess the consistency of a measure from one time to another –Administer the same test to the same (or a similar) sample on two different occasions

Types of reliability  Parallel-Forms Reliability –Used to assess the consistency of the results of two-tests constructed in the same way from the same content domain –Administer both instruments to the same sample  Internal Consistency reliability –Used to assess the consistency of results across items within a test –Calculate the average correlation of the items splitting into half [e.g., Cronbach’s alpha (α)]

Cronbach’s alpha (α) Trochim, 2002 Step 1

Cronbach’s alpha (α) Trochim, 2002 Step 2

Issues in educational research  Validity –Refers to what we measure is what we intend to measure (highly related to the research question) –We have to make sure that the tools we use to measure the variables are valid  Internal validity –The degree to which correct conclusions about causal relations can be drawn

Reliability and validity Trochim, 2002

Class activity  Discuss with your group members to see whether the issues facing by quantitative and qualitative research are the same in terms of reliability and validity

Issues in educational research  Applicability –To be of any use educational research must generalize from the sample to the population –The findings of the research should be able to inform teaching practice or policy formulation

Issues in educational research  Feasibility –There may be some excellent ideas in educational research but due to some reasons, the research may not be feasible –These reasons may include resource, time or ethical considerations

Issues in educational research  Significance –Educational research should have impact or acceptable level of importance to the field –Significance = the value of the research

Issues in educational research  Ethics –The research we conduct should not violate the ethical standard –For instance, it is not ethical to segregate a group of students during their programme in order to test out a new teaching technique

Common errors made in research  Selective observation  Inaccurate observation  Overgeneralization  Made-up information  Ex post facto hypothesizing  Illogical reasoning  Ego involvement in understanding  Premature closure of inquiry

Selective observation  It happens when attention is drawn to observations or answers that confirm pre- existing beliefs  A few ways you can try to avoid selective observation in your research: –Literature review –Decide your research approach beforehand –Take thorough notes –Watch for "disconfirmatory" information –Consider both "sides" of your study

Inaccurate observation  It occurs when we “misremember” or “misrecord” data  To increase your accuracy you have to plan your observation: –Using forms –Doing time/area sampling –Writing down as much information as you can –To practice observing and recording your observations before actually doing it for real

Overgeneralization  Overgeneralization is generalizing to others who are different from one's research population  To avoid overgeneralizing: –Replicate one's study –Support many tests of the same theory –Attempt to use representative samples –Recognize the limitations of one's research

Made-up information  It happens when one fills in details without a scientific basis for doing so  Instead of making up information, you could do a variety of things: –Ask someone who knows what's going on for their opinion –Do a thorough literature review to help you understand possible outcomes and unique customs –Rely on prior research studies and/or theory to guide your own interpretation of what was going on

Ex post facto hypothesizing  It occurs when a researcher decides what happened after it happened and after the study was done  In scientific research, we have to decide what will happen before we do our study, not after

Illogical reasoning  Illogical reasoning is just that: illogical  to avoid illogical reasoning in your own research: –Base your decisions on prior research and theory –Make extensive use of peer review

Ego involvement in understanding  It happens when people let the human side of them dictate their findings and how they view findings by other researchers  To avoid ego involvement: –To try to remain neutral –To stay away from topics about which you can't be neutral –To use the team approach

Premature closure of inquiry  It occurs when we decide that we know enough about a topic and decide that it no longer warrants future study  To deal with premature closure: –To keep looking for answers, even if it involves using different approaches –To find non-controversial ways to do it as far as possible

Mystification  It happens when we attribute results to the supernatural  To avoid mystification: –Keep looking for answers when you feel tempted to rely on enchanted explanations –Peer review will prevent you from embarrassing yourself

End of lesson 9