winnie mucherah ball state university FOUNDATIONS OF QUALITY RESEARCH DESIGN: RELIABILITY & VALIDITY

Literature review Systematic identification, location, and analysis of documents containing information related to the research problem Reviews are used to guide practice and/or to guide research Narrative reviews Topic reviews Theoretical reviews Meta-analyses (Mills, Airasian, & Gay, 2012)

Types of reviews Narrative/Traditional Reviews Most often conducted when writing dissertations and theses in the social sciences Also used in introductory paragraphs of a typical research article Provides a brief narrative about previous research on a subject to set the context for the current research topic

Topic reviews Introductory and investigatory reviews Conducted when working on a topic for the first time Often includes introductory works, e.g., encyclopedia entries and textbooks Criteria for good topic reviews: Recency (based on up-to-date sources) Importance (built on important sources, quality of the journal, impact factor) Breath (sources discuss topic broadly)

Theoretical reviews Not usually featured in lists of types of reviews, but are important subtypes It’s a version of a traditional/narrative review It’s specific purpose is to synthesize established theories by focusing on points of agreement and/or to generate new theories by focusing on gaps To either synthesize previous theories or to generate new ones.

Meta-analyses reviews Systematic reviews/ Research synthesis Systematic- used frequently to refer to evidence-based practical applications Research synthesis-often refers to research that is not necessarily tied to practical applications Similar: researcher states in advance the procedures for findings, selecting, coding and analyzing the data Data enables you to calculate effect size

Effect size Effect size is aptly named It’s a measure of the size of an effect. Specifically, it’s a standardized measure Standardized measures are often stated in standard deviation units Therefore, they can be used to compare and combine results across studies Comparing and combining results across studies is the whole point of meta-analysis.

quantitative v. qualitative Quantitative research Numerical data Ex - surveys and tests Research plan includes an introduction, method section, data analysis description, and results Qualitative research Comprehensive, narrative, and visual data Ex - interviews and naturalistic observations Research plan must be responsive to the context and setting under study Mixed-method design is ideal (Mills, Airasian, & Gay, 2012)

correlational v. Experimental Correlational research Collecting data to determine whether a relation exists between two or more quantifiable variables Measured by a correlation coefficient (r) Strength of relationship ranges from 0 to 1 Relationship can be positive or negative (inverse) Correlation is not causation

Experimental research Random assignment to groups Involves IV and DV At least one independent variable is manipulated Effect of one or more dependent variable(s) observed Quantitative measure of the degree of correspondence between two or more respondents

Reliability It’s the consistency or agreement among measures Consistency of data collection Results are more likely to be repeatable if you conduct the experiment all over again (because the sample size is large enough to produce the necessary precision) Reliability coefficients generally range from 1.0 for a perfectly reliable measure to 0 for one that is completely inconsistent from one rater/test/observation to the next Cronbach’s alpha (α)-estimates internal consistency (Rumsey, 2005)

Measure of reliability Cronbach’s alpha (α) It’s used when you want to know whether the items in your scale or index are measuring aspects of the same thing The “scale if item deleted” feature helps identify items that could be removed or analyzed individually (IRT).70 is usually considered the minimum acceptable level; higher levels are needed when results are used for high- stakes decisions

Types of reliability Inter-rater reliability-refers to the consistency of two or more raters Test-retest reliability-refers to the consistency of the same test over time or consistency of results on repeated tests Internal reliability- refers to the consistency of multiple questions probing aspects of the same concept

Validity It’s a central issue at all stages of a research project Chief concern is whether the study is set up so that you can reach justifiable conclusions about your topic. This is referred to as Internal Validity It addresses the question: Do my conclusions apply to my sample? The degree to which differences on a measure are attributable to the manipulation of the independent variable This is highest in true experimental studies (Mills, Airasian, & Gay, 2012)

External validity The degree to which results will be generalizable and to a certain extent replicable in other settings It addresses the question: Do my conclusions apply to anyone else? Can you generalize your conclusions beyond the participants in the experiment? The answer depends on the quality and the appropriateness of your sample Construct validity: are concepts measured in ways that enable us to study what we aim to study? Content validity: is the measure thorough or representative of the thing being measured?

Sampling procedures Population collection of all individuals of interests Sample subset of the population we measure Parameter a numeric characterization of the population that is of interest to us Statistic a numeric characterization of the sample that is an estimate of the population Since we cannot access population, we don’t have access to parameter, so we take a sample we can obtain, then we make a numeric measurement, also known as a statistic Coladarci & Cobb, 2014

Contextualizing your research Refining the substantive question and developing a plan for collecting relevant data Use of existing/new measures: Use Factor Analysis FA helps you decide about reliability and validity of your measurements of latent variables and thus how to analyze and interpret them FA is simply correlations and associations among items Purpose of FA is to improve the measurement of latent variables or constructs that cannot be directly observed (Coladarci & Cobb, 2014)

Latent variables Latent variables can only be studied indirectly by using indicators of observed variables, e.g., in a multi-item measure of traits, the items would be indicators (or observed variables) and clusters of questions identified by the FA would help you identify the factors or latent variables, which are the constructs or concepts you seek in your research. E.g., 15 questions toward a controversial issue Efficacy or social tolerance or attitudes

Types of Factor analysis Exploratory FA and Confirmatory FA EFA-used when researchers are looking for interesting patterns among variables CFA-used when researchers have theories about the patterns they want to test The two are often linked because it is very common to conduct them in sequence-first EFA to refine theories, then CFA to test them.

Conclusion Substantive Question ---> Statistical Question ---> Statistical Conclusion -- -> Substantive Conclusion Substantive Conclusion is a context- based conclusion

references Coladarci, T., Cobb, C.D., Minium, E.W. & Clarke, R.C. Fundamentals of Statistical Reasoning in Education. Mills, G.E., Airasian, P. & Gay, L.R Educational Research: Competencies for analysis and applications. 10 th Edition. Rumsey, D Statistics Workbook for Dummies.