1. PSYC512: Research Methods PSYC512: Research Methods Lecture 7 Brian P. Dyre University of Idaho

2. PSYC512: Research Methods Lecture 7 Outline Questions about material covered in Lecture 6 Questions about material covered in Lecture 6 Measures: scales and sensitivity Measures: scales and sensitivity More on Measurement More on Measurement Reliability, Precision, and Validity Reliability, Precision, and Validity Hypothesis testing and Variables Hypothesis testing and Variables Variables and Research Design Variables and Research Design Defining Variables Defining Variables

3. PSYC512: Research Methods Features of Measures: Reliability The ability of a measure to produce consistent results when repeated measurements are taken under identical conditions The ability of a measure to produce consistent results when repeated measurements are taken under identical conditions Types: Types: precision: physical measurement (1/noise) precision: physical measurement (1/noise) margin of error: sampling in surveys margin of error: sampling in surveys interrater reliability: observers viewing the same behavior interrater reliability: observers viewing the same behavior Test-retest, parallel forms and split-half reliabilities: psychological tests Test-retest, parallel forms and split-half reliabilities: psychological tests

4. PSYC512: Research Methods Other Features of Measures Accuracy Accuracy does a measure produce results that agree with a known standard? does a measure produce results that agree with a known standard? Accuracy vs. Precision Accuracy vs. Precision Validity Validity Measurement validity: the extent to which your measure indeed measures what it is intended to measure Measurement validity: the extent to which your measure indeed measures what it is intended to measure Types: Face validity, Content validity, Criterion- related validity (concurrent vs. predictive), Construct validity Types: Face validity, Content validity, Criterion- related validity (concurrent vs. predictive), Construct validity Relationship between reliability and validity Relationship between reliability and validity

5. PSYC512: Research Methods Hypothesis Testing: Variables Hypothesis testing is the process by which hypothetical relationships between variables (something that varies in quantity or quality) are assessed (the relationships are deduced from one or more theories) Hypothesis testing is the process by which hypothetical relationships between variables (something that varies in quantity or quality) are assessed (the relationships are deduced from one or more theories) Types of variables Types of variables Dependent variable  measure Dependent variable  measure Independent variable  manipulation Independent variable  manipulation Extraneous variable  not pertinent to hypotheses Extraneous variable  not pertinent to hypotheses Confounding variable  extraneous variable that covaries with your manipulated variable (typically we try to control these to eliminate the covariance) Confounding variable  extraneous variable that covaries with your manipulated variable (typically we try to control these to eliminate the covariance) Intervening variable  theoretical construct of interest that is not directly observable (e.g., group cohesiveness, mental workload) Intervening variable  theoretical construct of interest that is not directly observable (e.g., group cohesiveness, mental workload)

6. PSYC512: Research Methods Variables and Research Designs Relationships can be hypothesized between Relationships can be hypothesized between Multiple dependent measures  correlational research design: presence or absence of a relation between the variables can be tested, but not causality Multiple dependent measures  correlational research design: presence or absence of a relation between the variables can be tested, but not causality Manipulated (independent) variables and some measure  experimental design, with proper control of confounding variables (e.g., random assignment to experimental treatment groups) causality may be inferred Manipulated (independent) variables and some measure  experimental design, with proper control of confounding variables (e.g., random assignment to experimental treatment groups) causality may be inferred

7. PSYC512: Research Methods Defining Variables: Operationism Operationism: psychological concepts are equivalent to the operations (manipulations or measures) used to define those concepts Operationism: psychological concepts are equivalent to the operations (manipulations or measures) used to define those concepts Hunger: the state produced by food deprivation Hunger: the state produced by food deprivation Only observable operations are included in theoretical or hypothetical statements Only observable operations are included in theoretical or hypothetical statements You cannot separate the concept from its operations—cannot generalize, concept has no external validity You cannot separate the concept from its operations—cannot generalize, concept has no external validity

8. PSYC512: Research Methods Defining Variables: Converging Operations or Network Specification Multiple operations or a set of operations can be used to define a concept, not just one Multiple operations or a set of operations can be used to define a concept, not just one Operations can converge to scientifically isolate intervening variables through a process of converging operations (Garner, Hake, & Eriksen, 1956) Operations can converge to scientifically isolate intervening variables through a process of converging operations (Garner, Hake, & Eriksen, 1956) selective influence – experimental manipulations affect particular intervening variables but not others selective influence – experimental manipulations affect particular intervening variables but not others convergence – different operations can be used to manipulate or measure a common intervening variable or psychological construct convergence – different operations can be used to manipulate or measure a common intervening variable or psychological construct

9. PSYC512: Research Methods Converging Operations Example: The phenomenon of “Perceptual” Defense (Garner, Hake, & Eriksen, 1956) Example: The phenomenon of “Perceptual” Defense (Garner, Hake, & Eriksen, 1956) Two Possibilities Two Possibilities perceptual discrimination of vulgar words takes longer perceptual discrimination of vulgar words takes longer responding with a vulgar word takes longer responding with a vulgar word takes longer Operationist: perception is the discrimination response, therefore, we can’t tell which Operationist: perception is the discrimination response, therefore, we can’t tell which Converging operations: add a second, orthogonal operation—exchange the vulgar and neutral response mappings Converging operations: add a second, orthogonal operation—exchange the vulgar and neutral response mappings

10. PSYC512: Research Methods Network Specification of Meaning Psychological Concepts are defined by their relations with other concepts rather than a unitary operational definition Psychological Concepts are defined by their relations with other concepts rather than a unitary operational definition Introduction and Discussion sections of papers describe the relationships of our variables to all other relevant variables and concepts—what G, H, & E call assumed operations Introduction and Discussion sections of papers describe the relationships of our variables to all other relevant variables and concepts—what G, H, & E call assumed operations Method and results sections describe the specific converging operations we use Method and results sections describe the specific converging operations we use

11. PSYC512: Research Methods Construct Validity The soundness of our operations, do they manipulate or measure the intervening variable that they are intended to manipulate or measure? The soundness of our operations, do they manipulate or measure the intervening variable that they are intended to manipulate or measure? Types (Campbell & Fiske, 1959) Types (Campbell & Fiske, 1959) Discriminant validation: operation should not affect or correlate with operations on other intervening variables Discriminant validation: operation should not affect or correlate with operations on other intervening variables Convergent validation: operation should affect or correlate with other operations on the same intervening variable Convergent validation: operation should affect or correlate with other operations on the same intervening variable

12. PSYC512: Research Methods Testing Hypotheses Hypothesis testing is the process by which hypothetical relationships between intervening variables are assessed Hypothesis testing is the process by which hypothetical relationships between intervening variables are assessed Hypotheses are always tested relative to one-another or to a “null” hypothesis Hypotheses are always tested relative to one-another or to a “null” hypothesis Examples Examples Comparing Groups Comparing Groups Assessing Performance Interventions Assessing Performance Interventions Assessing Relationships between variables Assessing Relationships between variables Problem: Measurement Noise Problem: Measurement Noise

13. PSYC512: Research Methods Hypothesis Testing: Probability and Statistics Why are probability and statistics important? Why are probability and statistics important? Used to assess variability in a measure Used to assess variability in a measure Effect (treatment) Variance Effect (treatment) Variance Variability due to relationship between variables or effect of different levels of independent variable (treatments) Variability due to relationship between variables or effect of different levels of independent variable (treatments) “Good” variance that we want to maximize “Good” variance that we want to maximize Error Variance Error Variance Variability in measure due to factors other than the treatment Variability in measure due to factors other than the treatment “Bad” variance that we want to minimize “Bad” variance that we want to minimize Probability and Statistics are simply tools used to assess (descriptive statistics) and compare (inferential statistics) these sources of variability Probability and Statistics are simply tools used to assess (descriptive statistics) and compare (inferential statistics) these sources of variability

14. PSYC512: Research Methods Visualizing Variability: Distributions of Frequency and the Histogram Histograms: used to represent frequencies of data in different classes or categories Histograms: used to represent frequencies of data in different classes or categories BinFrequency 00 10 20 30 43 51 66 74 82 96 101

15. PSYC512: Research Methods Displaying Histograms: Stem and Leaf Plots Stem and Leaf plots are used to display histograms graphically (on their side) using only typed characters Stem and Leaf plots are used to display histograms graphically (on their side) using only typed characters StemLeaf(hypothetical histogram for IQ) 678 735668 8012234445555667777889 900011233333334445566667889999 1001112233334444445566677777888899 1001112233334444445566677777888899 110001122233444566777899 110001122233444566777899 120012569 120012569 1302 1302

16. PSYC512: Research Methods Distributions of Probability Density Similar to frequency histogram except y-axis now represents probability density (mass) rather than frequency Similar to frequency histogram except y-axis now represents probability density (mass) rather than frequency Probability density = Frequency/N Probability density = Frequency/N

17. PSYC512: Research Methods Some Types of Distributions NormalGamma

18. PSYC512: Research Methods Measures of the Center of a Distribution Measures of center represent the general magnitude of scores in a distribution Measures of center represent the general magnitude of scores in a distribution Mode: most frequent score Mode: most frequent score Median: the middle score of an ordered distribution Median: the middle score of an ordered distribution Mean (average):where X is the data and Mean (average):where X is the data and N is the total number of observations

19. PSYC512: Research Methods Measures of the Spread of a Distribution Measures of spread are used to assess the consistency of scores in a distribution Measures of spread are used to assess the consistency of scores in a distribution Range = max score – min score Range = max score – min score Interquartile range = score(Q3) – score(Q1) Interquartile range = score(Q3) – score(Q1) Variance (  2 ) and standard deviation (  ) Variance (  2 ) and standard deviation (  ) where X is the data, m is the mean of the data, and N is the total number of observations

20. PSYC512: Research Methods More on Variance Standard Deviation (  ) = sqrt(variance) Standard Deviation (  ) = sqrt(variance) where X is the data, m is the mean of the data, and N is the total number of observations Why N instead of N-1? Populations vs. Samples Remembering how to compute variance Remembering how to compute variance “the mean of the squares – square of the means”

21. PSYC512: Research Methods Describing Distributions Parametrically: Statistical Moments Any distribution based on interval or ratio data can be summarized by its statistical moments Any distribution based on interval or ratio data can be summarized by its statistical moments First Moment: Mean—location of distribution on x-axis First Moment: Mean—location of distribution on x-axis Second Moment: Variance—dispersion of distribution Second Moment: Variance—dispersion of distribution Third Moment: Skewness—symmetry of distribution Third Moment: Skewness—symmetry of distribution Fourth Moment: Kurtosis—degree of “peakedness” Fourth Moment: Kurtosis—degree of “peakedness”

22. PSYC512: Research Methods Estimators Sample statistics estimate population parameters Sample statistics estimate population parameters Mean: M or vs.  Mean: M or vs.  Variance: s 2 vs.  2 Variance: s 2 vs.  2 Properties of Estimators Properties of Estimators Sufficiency: uses all information in sample (mean and variance are sufficient, mode and range are not) Sufficiency: uses all information in sample (mean and variance are sufficient, mode and range are not) Unbiasedness: expected value approaches real value with increased sampling Unbiasedness: expected value approaches real value with increased sampling Efficiency: tightness of cluster of sample statistics relative to the population parameter Efficiency: tightness of cluster of sample statistics relative to the population parameter Resistance: influence of outliers on sample statistic Resistance: influence of outliers on sample statistic

23. PSYC512: Research Methods Next Time… Topic: Research Designs and Inferential Statistics Topic: Research Designs and Inferential Statistics Be sure to: Be sure to: Read the assigned readings (Howell chapters 6-7) Read the assigned readings (Howell chapters 6-7) Continue searching and reading the scientific literature for your proposal Continue searching and reading the scientific literature for your proposal