1. The Theory of Sampling and Measurement

2. Sampling First step in implementing any research design is to create a sample. First step in implementing any research design is to create a sample. We cannot study the theoretical population of all conceivable events (e.g., events that have not occurred), nor can we usually study all instances of actual events. We select some instances to study and not others. Those we include are our sample. We cannot study the theoretical population of all conceivable events (e.g., events that have not occurred), nor can we usually study all instances of actual events. We select some instances to study and not others. Those we include are our sample. How our sample is selected is critical for external validity or generalizability. How our sample is selected is critical for external validity or generalizability.

3. Who do you want to generalize to? Groups in Sampling

4. The theoretical population

5. What population can you get access to? Groups in Sampling The theoretical population

6. Groups in Sampling The Theoretical Population The study population

7. How can you get access to them? Groups in Sampling The theoretical population The study population

8. Groups in Sampling The theoretical population The study population The sampling frame

9. Who is in your study? Groups in Sampling The theoretical population The study population The sampling frame

10. Groups in Sampling The theoretical population The study population The sampling frame The sample

11. Types of Samples Probability Sampling Probability Sampling Simple random Simple random Stratified random Stratified random Cluster or area random Cluster or area random Non-Probability Sampling Accidental Modal instance Expert Snowball Case study (intentional selection)

12. The Sampling Distribution AverageAverageAverage 4.44.24.03.83.63.43.23.0 15 10 5 0 The sampling distribution......is the distribution of a statistic across an infinite number of samples. Sample 4.44.24.03.83.63.43.23.0 5 0 5 0 Sample 4.44.24.03.83.63.43.23.0 5 0 5 0 Sample 4.44.24.03.83.63.43.23.0 5 0 5 0

13. Population Parameter 4.54.03.53.0 150 100 50 0 Self esteem Frequency The population has a mean of 3.75......and a standard unit of.25. This means About 64% of cases fall between 3.5 - 4.0. About 95% of cases fall between 3.25 - 4.25. about 99% of cases fall between 3.0 - 4.5

14. Sampling Distribution 4.54.03.53.0 150 100 50 0 Self-esteem Frequency The population has a mean of 3.75.

15. Sampling Distribution 4.54.03.53.0 150 100 50 0 Self-esteem Frequency The population has a mean of 3.75......and a standard error of.25.

16. Inferring Population from Sample 4.54.03.53.0 150 100 50 0 Self esteem Frequency The sample has a mean of 3.75......and a standard deviation of.25. This means 64% chance true population mean falls between 3.5 - 4.0. 95% chance true population mean falls between 3.25 - 4.25. 99% chance true population mean falls between 3.0 - 4.5

17. Figure 3.4 Labor Repression and Growth in the Asian Cases, 1970-1981

18. Figure 3.5 Labor Repression and Growth in the Full Universe of Developing Countries,1970-1981

19. Measurement Operationalization is the process of translating theoretical constructs into observable indicators. Operationalization is the process of translating theoretical constructs into observable indicators. Construct validity and reliability are the criteria we use to evaluate how well you have operationalized your concepts. Construct validity and reliability are the criteria we use to evaluate how well you have operationalized your concepts. Both matter regardless of the level of measurement and whether you are using qualitative or quantitative indicators. Both matter regardless of the level of measurement and whether you are using qualitative or quantitative indicators.

20. The Hierarchy of Levels Nominal Interval Ratio Attributes are only named; weakest Attributes can be ordered Distance is meaningful Absolute zero Ordinal

21. Nominal Measurement The values “name” the attribute uniquely. The values “name” the attribute uniquely. The name does not imply any ordering of the cases. The name does not imply any ordering of the cases.

22. Ordinal Measurement When attributes can be rank-ordered… Distances between attributes do not have any meaning. Distances between attributes do not have any meaning.

23. Interval Measurement When distance between attributes has meaning, for example, temperature (in Fahrenheit) -- distance from 30-40°F is same as distance from 70-80°F Note that ratios don’t make any sense -- 80°F is not twice as hot as 40°F. Note that ratios don’t make any sense -- 80°F is not twice as hot as 40°F.

24. Ratio Measurement Has an absolute zero that is meaningful Has an absolute zero that is meaningful Can construct a meaningful ratio (fraction), for example, number of clients in past six months Can construct a meaningful ratio (fraction), for example, number of clients in past six months

25. Construct Validity Key problem is that we have abstract theoretical construct – power, democracy, development, corruption, etc. – that we can never observe directly. Key problem is that we have abstract theoretical construct – power, democracy, development, corruption, etc. – that we can never observe directly. Yet, to test propositions requires that we have some indicator for the construct – or at least have proxies that we can argue are capturing some attributes of the construct. Yet, to test propositions requires that we have some indicator for the construct – or at least have proxies that we can argue are capturing some attributes of the construct. Our indicator is an analogy (to an analogy). Our indicator is an analogy (to an analogy).

26. Assessing Construct Validity Translation Validity Translation Validity Face Validity: plausible on its “face” Face Validity: plausible on its “face” Content Validity: matches lists of attributes Content Validity: matches lists of attributes Criterion-related Validity Criterion-related Validity Predictive Validity: predicts accurately Predictive Validity: predicts accurately Concurrent Validity: distinguishes appropriately between groups Concurrent Validity: distinguishes appropriately between groups Convergent Validity Convergent Validity Discriminant Validity Discriminant Validity

27. The Convergent Principle Alternative measures of a construct should be strongly correlated.

28. How It Works Theory Self-esteemconstruct Item 1 Item 2 Item 3 Item 4 You theorize that the items all reflect self-esteem.

29. How It Works Theory Observation Self-esteemconstruct Item 1 Item 2 Item 3 Item 4 1.00.83.89.91.831.00.85.90.89.851.00.86.91.90.861.00 The correlations provide evidence that the items all converge on the same construct.

30. Convergent Validity in Measures of “Democracy” 1985 | polity2 pollib civlib reg 1985 | polity2 pollib civlib reg-------------+------------------------------------ polity2 | 1.0000 -0.9148 -0.8770 -0.8601 polity2 | 1.0000 -0.9148 -0.8770 -0.8601 pollib | -0.9148 1.0000 0.9176 0.8440 pollib | -0.9148 1.0000 0.9176 0.8440 civlib | -0.8770 0.9176 1.0000 0.8053 civlib | -0.8770 0.9176 1.0000 0.8053 reg | -0.8601 0.8440 0.8053 1.0000 reg | -0.8601 0.8440 0.8053 1.0000

31. Convergent Validity in Measures of “Education” 1985 | 123456 -------------+------------------------------------------------------ Ed. spending | 1.0000 -0.1217 0.2415 0.3563 0.0214 0.0195 Illiteracy (%) | -0.1217 1.0000 -0.5797 -0.7306 -0.8569 -0.6196 Cohort to Grade 4 | 0.2415 -0.5797 1.0000 0.4419 0.6553 0.3654 % Grade School | 0.3563 -0.7306 0.4419 1.0000 0.6230 0.3612 % Secondary School | 0.0214 -0.8569 0.6553 0.6230 1.0000 0.7576 % College | 0.0195 -0.6196 0.3654 0.3612 0.7576 1.0000

32. The Discriminant Principle Measures of different constructs should not correlate highly with each other.

33. How It Works Theory Self-esteemconstruct SE 1 SE 2 Locus-of-controlconstruct LOC 1 LOC 2

34. How It Works Theory Self- esteem construct SE 1 SE 2 Locus-of-controlconstruct LOC 1 LOC 2 You theorize that you have two distinguishable constructs.

35. How It Works Theory Self-esteemconstruct SE 1 SE 2 Locus-of-controlconstruct LOC 1 LOC 2 Observation r SE 1, LOC 1 =.12 r SE 1, LOC 2 =.09 r SE 2, LOC 1 =.04 r SE 2, LOC 2 =.11 The correlations provide evidence that the items on the two tests discriminate.

36. Theory Self-esteemconstruct SE 1 SE 2 SE 3 Locus-of-controlconstruct LOC 1 LOC 2 LOC 3 We have two constructs. We want to measure self-esteem and locus of control. For each construct, we develop three scale items; our theory is that items within the construct will converge and Items across constructs will discriminate.

37. Theory Observation Self-esteemConstruct SE 1 SE 2 SE 3 Locus-of-controlconstruct LOC 1 LOC 2 LOC 3 1.00.83.89.02.12.09.831.00.85.05.11.03.89.851.00.04.00.06.02.05.041.00.84.93.12.11.00.841.00.91.09.03.06.93.911.00 SE 1 SE 2 SE 3 LOC 1 LOC 2 LOC 3 SE 1 SE 2 SE 3 LOC 1 LOC 2 LOC 3 Green and red correlations are Convergent; yellow are Discriminant.

38. Theory Observation Self-esteemconstruct SE 1 SE 2 SE 3 Locus-of-controlconstruct LOC 1 LOC 2 LOC 3 1.00.83.89.02.12.09.831.00.85.05.11.03.89.851.00.04.00.06.02.05.041.00.84.93.12.11.00.841.00.91.09.03.06.93.911.00 SE 1 SE 2 SE 3 LOC 1 LOC 2 LOC 3 SE 1 SE 2 SE 3 LOC 1 LOC 2 LOC 3 The correlations support both convergence and discrimination, and therefore construct validity.

39. What Is Reliability? The “repeatability” of a measure The “repeatability” of a measure The “consistency” of a measure The “consistency” of a measure The “dependability” of a measure The “dependability” of a measure

40. True Score Theory 1234512345 1234512345 3Scan a multitude of information and decide what is important. 1234512345 1234512345 1234512345 1234512345 1234512345 1Manage time effectively 2Manage resources effectively. 3Scan a multitude of information and decide what is important. 4Decide how to manage multiple tasks. 5Organize the work when directions are not specific. 1Manage time effectively Rating Sheet Observedscore = Trueability + Randomerror T e + X

41. The Error Component T e + X Two components: Random error Random error Systematic error Systematic error erererer eseseses

42. The Revised True Score Model T erererer + X eseseses +

43. Random Error X Frequency The distribution of X with no random error

44. Random Error X Frequency The distribution of X with no random error The distribution of X with random error Notice that random error doesn’t affect the average, only the variability around the average.

45. Systematic Error X Frequency The distribution of X with no systematic error

46. Systematic Error X Frequency The distribution of X with no systematic error The distribution of X with systematic error Notice that systematic error does affect the average; we call this a bias.

47. If a Measure Is Reliable... X1X1X1X1 X2X2X2X2 We should see that a person’s score on the same test given twice is similar (assuming the trait being measured isn’t changing).

48. If a Measure Is Reliable... X1X1X1X1 X2X2X2X2 T + e 1 T + e 2 Recall from true score theory that... But, if the scores are similar, why are they similar?

49. If a Measure Is Reliable... X1X1X1X1 X2X2X2X2 T + e 1 T + e 2 The only thing common to the two measures is the true score, T. Therefore, the true score must determine the reliability.

50. Reliability Is... a ratio variance of the true scores variance of the measure var(T) var(X)

51. Reliability Is... a ratio variance of the true scores variance of the measure We can measure the variance of the observed score, X. The greater the variance, the less reliable the measure.

52. This Leads Us to... We cannot calculate reliability exactly; we can only estimate it. We cannot calculate reliability exactly; we can only estimate it. Each estimate attempts to capture the consequences of the true score in different ways. Each estimate attempts to capture the consequences of the true score in different ways.

53. We want both Reliability and Validity

54. Reliability and Validity Reliable but not valid

55. Reliability and Validity Valid but not reliable

56. Reliability and Validity Neither reliable nor valid

57. Reliability and Validity Reliable and valid

58. Assignment #1 Assess the validity and reliability of the IRIS-3 International Country Risk Guide. Assess the validity and reliability of the IRIS-3 International Country Risk Guide. Can examine a single instance, compare instances, analyze the full variation in the dataset, compare with additional measures, or use any other form of assessment. May use outside sources of data, history, or analysis (but document). Can examine a single instance, compare instances, analyze the full variation in the dataset, compare with additional measures, or use any other form of assessment. May use outside sources of data, history, or analysis (but document). The only restriction is that the paper must be empirical and examine issues of validity and reliability. The only restriction is that the paper must be empirical and examine issues of validity and reliability. 3-5 pages. Be concise. 3-5 pages. Be concise. Due Monday 10/24 at beginning of class. Due Monday 10/24 at beginning of class.