1. ASSESSMENT OF MORAL REASONING AT TWO FAITH BASED INSTITUTIONS David W. Kale, Ph.D. Director of Assessment, Mount Vernon Nazarene University dkale@mvnu.edu Joel Frederickson, Ph.D. Chair, Psychology Dept., Bethel University frejoe@bethel.edu

2. ASSESSMENT OF MORAL REASONING BETHEL UNIVERSITY

3. Moral Thinking: Using the DIT-2 as a Measure of Cognitive Development

4. What does the DIT-2 measure? A revised measure of moral reasoning based on Kohlberg’s theory of moral development. A revised measure of moral reasoning based on Kohlberg’s theory of moral development. Focus is on schemas NOT stages. Focus is on schemas NOT stages. Five scenarios (original DIT had six). Five scenarios (original DIT had six). Strong correlations between DIT-1 and DIT-2 (r=.79). Strong correlations between DIT-1 and DIT-2 (r=.79). Example Dilemma from original DIT: Heinz & the drug Example Dilemma from original DIT: Heinz & the drug

5. Schema Scores Personal Interest Schema Score: The proportion of items selected that appeal to stage 2 & 3 thinking. Stage 2 focuses on the direct advantages to the actor and on the fairness of simple exchanges of favor for favor. Stage 3 focuses on the good or evil intentions of the parties; concern for maintaining friendships and approval. Personal Interest Schema Score: The proportion of items selected that appeal to stage 2 & 3 thinking. Stage 2 focuses on the direct advantages to the actor and on the fairness of simple exchanges of favor for favor. Stage 3 focuses on the good or evil intentions of the parties; concern for maintaining friendships and approval.

6. Schema Scores Maintaining Norms Schema Score. The proportion of items that appeal to stage 4 thinking. Focus on maintaining existing legal system, existing roles and organizational structure. Maintaining Norms Schema Score. The proportion of items that appeal to stage 4 thinking. Focus on maintaining existing legal system, existing roles and organizational structure.

7. Schema Scores Postconventional Schema Score (P score) Focus on organizing society by appealing to consensus-producing procedures (majority vote), insisting on due process, and safeguarding basic rights. Organizing social arrangements & relationships in terms of intuitively appealing ideals. Postconventional Schema Score (P score) Focus on organizing society by appealing to consensus-producing procedures (majority vote), insisting on due process, and safeguarding basic rights. Organizing social arrangements & relationships in terms of intuitively appealing ideals.

8. Example Famine Scenario: Read through this scenario and the accompanying “issues” related to the dilemma. Famine Scenario: Read through this scenario and the accompanying “issues” related to the dilemma. Place the “issues” into the categories they are measuring Place the “issues” into the categories they are measuring –Personal Interests –Maintaining Norms –Postconventional Thinking –Meaningless/pretentious items

9. New Index (N2) N2 score is a new index and is considered superior to the P score (post conventional reasoning score). N2 score is a new index and is considered superior to the P score (post conventional reasoning score). There are two parts to this score: the degree to which respondents support post conventional responses (P score), plus the degree to which personal interest items receive lower ratings. There are two parts to this score: the degree to which respondents support post conventional responses (P score), plus the degree to which personal interest items receive lower ratings. Essentially, the N2 is a score that reflects the degree to which respondents will reject bad arguments (personal interest arguments). Essentially, the N2 is a score that reflects the degree to which respondents will reject bad arguments (personal interest arguments).

10. Validity & Reliability of DIT Differentiation of various age/education groups: 30% to 50% of the variance in DIT scores is attributable to level of education. Differentiation of various age/education groups: 30% to 50% of the variance in DIT scores is attributable to level of education.

11. Education and Moral Thinking As education level goes up, degree of postconventional thinking increases. As education level goes up, degree of postconventional thinking increases.

12. Validity & Reliability of DIT Differentiation of various age/education groups: 30% to 50% of the variance in DIT scores is attributable to level of education. Differentiation of various age/education groups: 30% to 50% of the variance in DIT scores is attributable to level of education. Longitudinal gains: Reviews of a dozen studies of Freshman to Senior college students show effect sizes of.80, one of the most dramatic effects of college. Longitudinal gains: Reviews of a dozen studies of Freshman to Senior college students show effect sizes of.80, one of the most dramatic effects of college.

13. Validity & Reliability of DIT Sensitive to moral education interventions Sensitive to moral education interventions Reliability is adequate (Test-retest and Cronbach alphas in upper.70s, lower.80s). Reliability is adequate (Test-retest and Cronbach alphas in upper.70s, lower.80s).

14. Use of the DIT at Bethel for Assessment Look at cross-sectional and longitudinal differences from Freshman to Senior year. Look at cross-sectional and longitudinal differences from Freshman to Senior year. We have also analyzed these Freshman to Senior changes by department. We have also analyzed these Freshman to Senior changes by department. Example: Business department typically had low growth. They made some changes to the curriculum (and faculty) and saw better growth in moral reasoning. Example: Business department typically had low growth. They made some changes to the curriculum (and faculty) and saw better growth in moral reasoning.

15. Use of the DIT at Bethel for Assessment Usually we see typical growth from Freshman to Senior year (.70-.80 effect sizes). Usually we see typical growth from Freshman to Senior year (.70-.80 effect sizes). A small cohort in our degree completion Business Management program saw similar growth. A small cohort in our degree completion Business Management program saw similar growth. Just beginning to get longitudinal data for our MBA program. Just beginning to get longitudinal data for our MBA program.

16. Use of the DIT at Bethel for Assessment Colleague uses pre/post DIT in his course “Being Just in an Unjust World” (essentially a course in Moral Thinking). Typically sees a big change for a semester (about.70 effect size). Colleague uses pre/post DIT in his course “Being Just in an Unjust World” (essentially a course in Moral Thinking). Typically sees a big change for a semester (about.70 effect size). This is the amount of change you would typically see in 4 years! This is the amount of change you would typically see in 4 years!

17. Pre/post DIT N2 Scores by Type of Course

18. Use of the DIT at Bethel for Assessment A small subset of students who had taken this course (N=23) took the DIT again four months later. A small subset of students who had taken this course (N=23) took the DIT again four months later. Students not only maintained growth, but showed a marginally significant increase. Students not only maintained growth, but showed a marginally significant increase.

19. ASSESSMENT OF MORAL REASONING AT MOUNT VERNON NAZARENE UNIVERSITY COMBINING THE QUANTITATIVE WITH THE QUALITATIVE

20. QUALITATIVE ASSESSMENT The Potter Box The Potter Box Definition of situation Values Loyalties Principles Ralph B. Potter, “The Logic of Moral Argument” in Towards a Discipline of Social Ethics, ed. Paul Deats (Boston Univ. Press, 1972).

21. DEFINITION OF THE SITUATION The letter is the engineer’s personal and professional correspondence. The letter is the engineer’s personal and professional correspondence. The letter is privileged communication between the engineer and his attorney. The letter is privileged communication between the engineer and his attorney. What other aspects of the situation would you consider important to consider? What other aspects of the situation would you consider important to consider?

22. VALUES Privacy Privacy Truth Truth What other values do you see? What other values do you see? What is your highest value? What is your highest value?

23. PRINCIPLES Always tell the truth

24. PRINCIPLES Tell the truth Tell the truth Protect the privacy of others Protect the privacy of others

25. LOYALTIES To whom is moral duty owed? To whom is moral duty owed? - The 17 year old - The engineer - Who else has a stake in this situation that needs to be protected? To whom do you have the highest loyalty? To whom do you have the highest loyalty?

26. MORAL AND ETHICAL THEORIES Aristotle’s Golden Mean Aristotle’s Golden Mean Biblical concept of love Biblical concept of love Utilitarianism Utilitarianism Rawls’ Theory of Justice Rawls’ Theory of Justice Kant’s Categorical Imperative Kant’s Categorical Imperative

27. BUILDING A RUBRIC FOR STUDENT SELF ASSESSMENT After dealing with several case studies, I ask students to answer two questions anonymously. After dealing with several case studies, I ask students to answer two questions anonymously. 1. In what ways have you improved in your moral and ethical reasoning ability? 1. In what ways have you improved in your moral and ethical reasoning ability? 2. In what ways would you like to continue to improve? 2. In what ways would you like to continue to improve?

28. BUILDING A RUBRIC FOR STUDENT SELF ASSESSMENT I identified eight dimensions of moral reasoning in their answers. I identified eight dimensions of moral reasoning in their answers. 1. Using clearly worded arguments when I state my position; 1. Using clearly worded arguments when I state my position; 2. Making sure I have good support for my position (facts, reasoning, statistics) 2. Making sure I have good support for my position (facts, reasoning, statistics) 3. Taking the views of others into consideration when I make my decision. 3. Taking the views of others into consideration when I make my decision. 4. Supporting my arguments with biblical truth. 4. Supporting my arguments with biblical truth.

29. BUILDING A RUBRIC FOR STUDENT SELF ASSESSMENT 5. Having a clear process for moving all the way to a decision; 5. Having a clear process for moving all the way to a decision; 6. Making my decisions based on reason rather than letting my emotions get the upper hand; 6. Making my decisions based on reason rather than letting my emotions get the upper hand; 7. Taking my time rather than rushing into a decision. 7. Taking my time rather than rushing into a decision. 8. Staying well informed on the issues of the day. 8. Staying well informed on the issues of the day.

30. BUILDING A RUBRIC FOR STUDENT SELF ASSESSMENT I then have students vote on the top five. I then have students vote on the top five. 1. Using clearly worded arguments when I state my position. 1. Using clearly worded arguments when I state my position. 2. Making sure I have good support for my position (facts, reasoning, statistics). 2. Making sure I have good support for my position (facts, reasoning, statistics). 3. Taking the views of others into consideration. 3. Taking the views of others into consideration. 4. Supporting my position with biblical truth. 4. Supporting my position with biblical truth. 5. Taking my time rather than rushing into a decision. 5. Taking my time rather than rushing into a decision.

31. BUILDING A RUBRIC FOR STUDENT SELF ASSESSMENT Building a rubric in this fashion is an assessment strategy in itself. Building a rubric in this fashion is an assessment strategy in itself. If there is an aspect of moral reasoning that does not turn up in their list that I think is important, that tells me either that they have not learned it or do not think it is nearly as important as I do. If there is an aspect of moral reasoning that does not turn up in their list that I think is important, that tells me either that they have not learned it or do not think it is nearly as important as I do.

32. QUANTITATIVE ASSESSMENT USING THE DIT-2 AT MVNU On the traditional side, we administer the DIT-2 to our incoming first year students, to our Juniors on Testing Day and to our seniors at the end of the Senior Colloquium course. On the traditional side, we administer the DIT-2 to our incoming first year students, to our Juniors on Testing Day and to our seniors at the end of the Senior Colloquium course. We also administer the DIT-2 to our nontraditional students at the beginning and at the end of their degree completion programs. We also administer the DIT-2 to our nontraditional students at the beginning and at the end of their degree completion programs.

33. N2 scores Trad. Juniors 26.51 26.51 Juniors nat’l. Av 32.65 32.65 p = <.01 p = <.01 Trad. Seniors 35.28 35.28 Seniors nat’l. Av 36.85 36.85 p =.079 p =.079 Trad. Juniors 26.51 26.51 Trad. Seniors 35.28 35.28 p = <.001 p = <.001 Trad. Juniors Trad. Juniors 26.51 26.51 Non-trad. Junior 23.06 23.06 p =.148 p =.148 Trad. Seniors 35.28 35.28 Non. Trad. Sen. 26.55 26.55 p =.0002 p =.0002

34. Trad. And Non-Trad. Students Post conventional Reason. Scores

35. SUMMARY There appears to be strong evidence from the use of the Defining Issues Test at two faith based institutions that a senior level Christian ethics course significantly improves students’ moral reasoning ability. There appears to be strong evidence from the use of the Defining Issues Test at two faith based institutions that a senior level Christian ethics course significantly improves students’ moral reasoning ability. Using both quantitative and qualitiative measures provides richer data as to exactly what aspects of students’ reasoning has improved. Using both quantitative and qualitiative measures provides richer data as to exactly what aspects of students’ reasoning has improved.