1. PSYC512: Research Methods PSYC512: Research Methods Lecture 16 Brian P. Dyre University of Idaho

2. PSYC512: Research Methods Lecture 16 Outline Relational Research Relational Research Observational Methods Observational Methods Assessing Reliability of observations Assessing Reliability of observations Sampling Sampling

3. PSYC512: Research Methods Relational (Non-Experimental) Research Observational Research Observational Research Naturalistic Observation Naturalistic Observation Ethnography Ethnography Sociometry Sociometry Case History Case History Archival Research Archival Research Surveys (self-reports) Surveys (self-reports) All of these methods require measurement of behaviors directly viewed, self-reported, or previously cataloged All of these methods require measurement of behaviors directly viewed, self-reported, or previously cataloged

4. PSYC512: Research Methods Establishing Reliable Measures for Relational Research Non-experimental research requires the definition of specific behavioral categories or recording units Non-experimental research requires the definition of specific behavioral categories or recording units Categories should be Categories should be based on hypotheses, informal observations, literature search based on hypotheses, informal observations, literature search developed before the behavior is observed or the archival data is analyzed developed before the behavior is observed or the archival data is analyzed simple and focused on specific behaviors or archival content simple and focused on specific behaviors or archival content exhaustive, mutually exclusive, independent exhaustive, mutually exclusive, independent

5. PSYC512: Research Methods Establishing Reliable Measures for Relational Research Behaviors within each category should be quantified using one or more of the following methods Behaviors within each category should be quantified using one or more of the following methods Frequency method: number of times behavior occurs Frequency method: number of times behavior occurs Duration method: how long a behavior lasts Intervals method: does behavior occur within discrete time intervals? Duration method: how long a behavior lasts Intervals method: does behavior occur within discrete time intervals? Behavior sequences: keep track of order of behavior in addition to frequency Behavior sequences: keep track of order of behavior in addition to frequency

6. PSYC512: Research Methods Establishing Reliable Measures for Relational Research Problem: Often behavior is complex and occurs too quickly to both observe and record at the same time Problem: Often behavior is complex and occurs too quickly to both observe and record at the same time Sampling methods Sampling methods Time: alternate observing and recording periods Time: alternate observing and recording periods Individual: observe and record only one individual at a time Individual: observe and record only one individual at a time Event: observe and record only one behavior at a time Event: observe and record only one behavior at a time Recording devices (video, audio) Recording devices (video, audio)

7. PSYC512: Research Methods Establishing Reliable Measures for Relational Research Problem: any single observer (or content analyzer) might be biased or their observations might be idiosyncratic or unreliable Problem: any single observer (or content analyzer) might be biased or their observations might be idiosyncratic or unreliable Solution: use multiple observers and quantify their differences by computing interrater reliability Solution: use multiple observers and quantify their differences by computing interrater reliability

8. PSYC512: Research Methods Establishing Reliable Measures for Non-Experimental Research Statistical methods for computing interrater reliability Statistical methods for computing interrater reliability Percent agreement = 100* Nagreements/Nobservations Percent agreement = 100* Nagreements/Nobservations Cohen’s Kappa (K) = (Po – Pc)/(1 – Pc) Cohen’s Kappa (K) = (Po – Pc)/(1 – Pc) Po is the actual agreement and Pc is the agreement you would expect by chance Po is the actual agreement and Pc is the agreement you would expect by chance Confusion Matrix observer 1 Confusion Matrix observer 1 angry loving angry15 observer 2 loving11 13 1326 Po = (cell11 + cell22)/N = (10 + 8)/26 Pc = (row1*col1 + row2*col2)/(N*N) = (15*13 + 11*13)/(26*26) 105 38

9. PSYC512: Research Methods Establishing Reliable Measures for Non-Experimental Research Statistical methods for computing interrater reliability Statistical methods for computing interrater reliability Pearson’s product-moment correlation Pearson’s product-moment correlation Observer bias Observer bias Blind observers Blind observers Objective vs. interpretive recording Objective vs. interpretive recording

10. PSYC512: Research Methods Sampling Why sample? We cannot usually measure the entire population of interest so we must rely on measuring a sample of the population Why sample? We cannot usually measure the entire population of interest so we must rely on measuring a sample of the population Goal of Sampling: to be able to generalize to everyone in the population of interest –sample must represent the population to insure external validity Goal of Sampling: to be able to generalize to everyone in the population of interest –sample must represent the population to insure external validity Terms Terms population: any group with size greater than 1 population: any group with size greater than 1 element: one member of a sample, e.g., person, family, city, country, etc. element: one member of a sample, e.g., person, family, city, country, etc. strata: sub-group of sample which is homogeneous with respect to some variable - e.g., male/female strata: sub-group of sample which is homogeneous with respect to some variable - e.g., male/female

11. PSYC512: Research Methods Random Sampling Techniques Simple random sample Simple random sample need an entire list (or access) to all elements of population need an entire list (or access) to all elements of population draw sample using names in drum, random number table, etc. draw sample using names in drum, random number table, etc. given a big enough sample it will be representative each member of population has an equal chance of being sampled. given a big enough sample it will be representative each member of population has an equal chance of being sampled. Systematic random sample (short cut) Systematic random sample (short cut) still need list of every element still need list of every element take every nth element, where n = pop size/sample size take every nth element, where n = pop size/sample size pick first element randomly pick first element randomly

12. PSYC512: Research Methods Stratified Sampling Techniques Stratified (Homogeneous Subgroup) Sample Stratified (Homogeneous Subgroup) Sample Proportional Stratified Sample Proportional Stratified Sample sample elements are in the same proportion as they occur in the population sample elements are in the same proportion as they occur in the population allows inferences from sample strata to population strata allows inferences from sample strata to population strata allows inferences from entire sample to entire population allows inferences from entire sample to entire population problem: small strata may not give enough detail problem: small strata may not give enough detail Equal Stratified Sample Equal Stratified Sample equal proportion of sample comes from each strata of population equal proportion of sample comes from each strata of population different size of strata populations - insures stability of sample from smaller strata different size of strata populations - insures stability of sample from smaller strata each strata is equally representative of its target population each strata is equally representative of its target population allows comparisons between strata - internally valid allows comparisons between strata - internally valid EXAMPLE: views of political parties in America EXAMPLE: views of political parties in America

13. PSYC512: Research Methods Other Sampling Techniques Purposeful Sample Purposeful Sample identify cluster of sample that is representative of entire population with respect to the variable of interest identify cluster of sample that is representative of entire population with respect to the variable of interest randomly select from cluster randomly select from cluster Incidental (convenience) Sample Incidental (convenience) Sample sample from convenient or available population e.g., subject pool! most psychological research does this sample from convenient or available population e.g., subject pool! most psychological research does this phone surveys – sample only people in phonebook phone surveys – sample only people in phonebook external validity is limited external validity is limited

14. PSYC512: Research Methods A Few Observations on Surveys (really, no pun intended) Good source of information: Dillman (1978) Mail and Telephone Surveys: The Total Design Method, New York: Wiley and Sons Good source of information: Dillman (1978) Mail and Telephone Surveys: The Total Design Method, New York: Wiley and Sons

15. PSYC512: Research Methods Next Time… Article Discussions Article Discussions