1. Reliability and Validity of Researcher-Made Surveys

2. Reliability

3. Authors should provide reliability coefficients of the scores for the data they analyze even when the focus of their research is not psychometric.

4. Reliability Authors should provide reliability coefficients of the scores for the data they analyze even when the focus of their research is not psychometric. Reliability is the amount of random fluctuation in individual scores.

5. Reliability Authors should provide reliability coefficients of the scores for the data they analyze even when the focus of their research is not psychometric. Reliability is the amount of random fluctuation in individual scores. In practice, score reliability is a matter of degree, because all scores include some random fluctuation.

6. Reliability It is the reliability of the data in hand in a given study that will drive study results, not the reliability of the scores described in the test manual. Reliability is the amount of random fluctuation in individual scores. In practice, score reliability is a matter of degree, because all scores include some random fluctuation.

7. Reliability It is the reliability of the data in hand in a given study that will drive study results, not the reliability of the scores described in the test manual. Poor score reliability may compromise … score validity. In practice, score reliability is a matter of degree, because all scores include some random fluctuation.

8. Reliability It is the reliability of the data in hand in a given study that will drive study results, not the reliability of the scores described in the test manual. Poor score reliability may compromise … score validity. Poor score reliability may compromise the ability of a study to yield noteworthy effects

9. Reliability Poor score reliability may compromise … score validity. Poor score reliability may compromise the ability of a study to yield noteworthy effects

10. Reliability Scores cant both measure nothing and measure something. Poor score reliability may compromise … score validity. Poor score reliability may compromise the ability of a study to yield noteworthy effects

11. Reliability Scores cant both measure nothing and measure something. Perfectly unreliable scores are perfectly random and cannot yield… significant results. Poor score reliability may compromise … score validity. Poor score reliability may compromise the ability of a study to yield noteworthy effects

12. Reporting Reliability Scores cant both measure nothing and measure something. Perfectly unreliable scores are perfectly random and cannot yield… significant results. Poor score reliability may compromise … score validity. Poor score reliability may compromise the ability of a study to yield noteworthy effects

13. Reporting Reliability

14. Reporting reliability coefficients for ones own data is the exception rather than the norm...Too few reliability estimates for analyzed data are provided in both journals…and doctoral dissertations.

15. Reporting Reliability Reporting reliability coefficients for ones own data is the exception rather than the norm...Too few reliability estimates for analyzed data both journals…andReporting reliability coefficients for ones own data is the exception rather than the norm...Too few reliability estimates for analyzed data are provided in both journals…and doctoral dissertations.

16. Reporting Reliability The most commonly used (reliability) estimate is Cronbachs (1951) coefficient alpha ( ).

17. number of items (sum of item variances) number of items-1 test variance Reporting Reliability 1-

18. The most commonly used (reliability) estimate is Cronbachs (1951) coefficient alpha ( ). Item score covariance plays an important role in both the numerator and the denominator of the estimate. Reporting Reliability

19. The most commonly used (reliability) estimate is Cronbachs (1951) coefficient alpha ( ). The intercorrelations of the items are the essential source of this kind of reliability. Reporting Reliability

20. Validity of Researcher-Made Surveys

21. Evidence of Validity

22. Patterns of Association

23. Evidence of Validity Patterns of Association Comparing Results from Different Versions of the Same Question

24. Evidence of Validity Patterns of Association Comparing Results from Different Versions of the Same Question Comparing Responses to Data from Other Sources

25. Evidence of Validity Patterns of Association Comparing Results from Different Versions of the Same Question Comparing Responses to Data from Other Sources Asking the Same Question Twice and Comparing Results

26. Patterns of Association Comparing Results from Different Versions of the Same Question Comparing Responses to Data from Other Sources Asking the Same Question Twice and Comparing Results Evidence of Validity Reliability

27. Patterns of Association Evidence of Validity

28. Patterns of Association Scores from different measures believed to measure similar things should correlate. Scores from different measures believed not to measure similar things should not correlate. Responses to items believed to represent the same dimensions or factors should correlate. Evidence of Validity

29. Comparing Responses to Data from Other Sources Evidence of Validity

30. Comparing Responses to Data from Other Sources Compare to records. Compare to physical testing. Compare to population estimates. Evidence of Validity

31. Face Validity of Survey Questions

32. Have a reason for every question you ask. Keep questions simple. Keep questions precise. Avoid leading questions. Foresee social desirability. Response options should be mutually exclusive and exhaustive. Provide temporal frame of reference. Use Likert format correctly.

33. Pilot Testing

34. Sample size >15 Discuss questions with respondents to find confusion or ambiguity. Pretest sample should resemble study sample. Examine variance among respondents. Refine answer options. Time how long it takes.

35. Group Assignment

36. Produce a Valid and Reliable Attitude or Psychological Scale in 90 Minutes

37. Write a 7 to 10 item scale.30 minutes Pilot test your items. 15 minutes Use another group. If necessary, revise your scale based on your pilot testing. Administer the revised scale to 10 minutes at least 6 people. Collect data. Enter data on SPSS. 20 minutes Compute coefficient alpha. Revise your scale based on reliability data. Report.15 minutes

38. Entering Survey Data on SPSS Item1Item2Item3 Survey1415 Survey2514 Survey3423 Item 1 I like salt. Strongly Disagree Disagree Neutral Agree Strongly Agree 1 2 3 4 5

39. Analyzing for Reliability in SPSS 1. Enter survey data. 2. Choose menu options:Analyze Scale Reliability Analysis 3. Choose and move variables to Items box. 4. Click Statistics…Click on: Item Scale Scale if Item Deleted 5. Read output. Identify items which have an Alpha if Item Deleted larger than the scales Alpha. Remove those items. 6. Re-run analysis with remaining items until satisfied.