1. Critical Thinking and Moral Reasoning Impact Assessment Tools for DE and ESD Nancy. L. Serrano, September 25 th 2009, University of Limerick

2. Presentation Outline Why a DE/ESD evaluation tool? Why a DE/ESD evaluation tool? Project background Project background Qualitative Vs. Quantitative Qualitative Vs. Quantitative Factors influencing decision-making on SD Factors influencing decision-making on SD Values and Attitudes Values and Attitudes Critical Thinking and DE/ESD Critical Thinking and DE/ESD Tool design parameters Tool design parameters Consultation day feedback Consultation day feedback Next steps… Next steps…

3. Why a DE/ESD Evaluation Tool? No standard assessment to measure impact of work in formal or non-formal DE/ESD sectors No standard assessment to measure impact of work in formal or non-formal DE/ESD sectors To increase transparency, communication, and possible comparisons To increase transparency, communication, and possible comparisons To allow for much-needed benchmarks and baselines to be established To allow for much-needed benchmarks and baselines to be established

4. Project Background July 2008 - April 2009 : RCE-Ireland Project “Situated Learning and DE/ESD Activism” - questionnaire tool developed testing 3 variables: situational, knowledge & ability, intrapersonal July 2008 - April 2009 : RCE-Ireland Project “Situated Learning and DE/ESD Activism” - questionnaire tool developed testing 3 variables: situational, knowledge & ability, intrapersonal (Hines, Homerford and Tomera, 1986) Focus on values and attitudes, and critical thinking - identified in literature for action-orientation, (IRISH AID, 1999, Tormey, 2003, Tilbury and Wortman, 2004) Focus on values and attitudes, and critical thinking - identified in literature for action-orientation, (IRISH AID, 1999, Tormey, 2003, Tilbury and Wortman, 2004) To develop a quantitative evaluation tool To develop a quantitative evaluation tool

5. Qualitative vs Quantitative Qualitative Interpretive Interpretive Individualistic Individualistic Time consuming Time consuming Wide-ranging Wide-ranging Non-comparative Non-comparative Complex to analyse Complex to analyse Not statistically significant Not statistically significant Can raise ‘new’ issues Can raise ‘new’ issuesQuantitative Addresses pre-determined questions Addresses pre-determined questions Straight forward to collate Straight forward to collate Statistically analytic Statistically analytic Comparative Comparative Dependent on evaluator knowledge Dependent on evaluator knowledge Does not encourage ‘new’ issues Does not encourage ‘new’ issues

6. Kirkpatrick’s Levels of Evaluation http://coe.sdsu.edu/eet/Articles/k4levels/index.htm Qualitative Quantitative

7. Factors influencing decision-making on sustainability Issues 1. Situational factors (e.g existence of recycle banks, public transport) 2. knowledge and ability factors (e.g. knowing about energy saving and how to do it) 3. intrapersonal factors (e.g. attitudes, values, acceptance of responsibility) (Hines, Hungerford and Tomera,1986) The intrapersonal factor MUST be dealt with for LASTING behavioural change

8. Values and Attitudes -Intrapersonal factors- Social Value Orientation Social Value Orientation Individualistic Individualistic Competitive Competitive Co-operative Co-operative (Osbaldiston and Sheldon, 2002, Komorita and Parks, 1995)

9. Critical Thinking and DE/ESD “Critical thinking is reasonable, reflective thinking that is focused on deciding what to believe or do.” “Critical thinking is reasonable, reflective thinking that is focused on deciding what to believe or do.” (Ennis, 1993) DE/ESD is cross-curricular DE/ESD is cross-curricular Some critical thinking skills are more relevant to DE/ESD e.g. Some critical thinking skills are more relevant to DE/ESD e.g. open-mindedness open-mindedness ability to judge credibility of sources ability to judge credibility of sources ask clarifying questions and draw conclusions ask clarifying questions and draw conclusions identify conclusions, reasons, and assumptions identify conclusions, reasons, and assumptions (Ennis, 1993) Critical thinking tests: general-content based, subject specific, multiple choice tests exist (Illinois Critical Thinking Project), essay-style tests (simulated situations) (The Ennis-Weir Critical Thinking Essay Test) Critical thinking tests: general-content based, subject specific, multiple choice tests exist (Illinois Critical Thinking Project), essay-style tests (simulated situations) (The Ennis-Weir Critical Thinking Essay Test)

10. Consultation Day Feedback Values + Attitudes Values + Attitudes Responsibility Responsibility Openness Openness Inter-connectedness Inter-connectedness Tolerance Tolerance Ownership Ownership Motivation Motivation Intention Intention Critical Thinking Skills Critical Thinking Skills See other perspectives See other perspectives Question+critique Question+critique Analyse complex issues Analyse complex issues Make informed choices Make informed choices Action planning Action planning Futures thinking Futures thinking Adapting to changing world Adapting to changing world

11. Evaluation Tool Design Format July 2008 - April 2009 : “Situated Learning and DE/ESD Activism” project - questionnaire tool, 3 variables: situational, knowledge & ability, intrapersonal July 2008 - April 2009 : “Situated Learning and DE/ESD Activism” project - questionnaire tool, 3 variables: situational, knowledge & ability, intrapersonal (Hines, Homerford and Tomera, 1986) Defining Issues Test (Bebeau and Thoma, 2003) Defining Issues Test (Bebeau and Thoma, 2003) moral reasoning psychometric test moral reasoning psychometric test Story-format or scenarios Story-format or scenarios participants rank statements in order of importance, in relation to decision made in each scenario participants rank statements in order of importance, in relation to decision made in each scenario based on the work of Lawrence Kohlberg. (Kohlberg, 1973). The DIT measures moral reasoning on a scale corresponding to these stages based on the work of Lawrence Kohlberg. (Kohlberg, 1973). The DIT measures moral reasoning on a scale corresponding to these stages Adapting scenarios model to test for values and attitudes and critical thinking skills Adapting scenarios model to test for values and attitudes and critical thinking skills Develop DE/ESD scale for action-orientation Develop DE/ESD scale for action-orientation Example Scenario question from original questionnaire Example Scenario question from original questionnaire

12. DE/ESD Scenario Decision Factors Influencing Factors tested Factors affecting decision-making in SD Personal EffectivenessValues & attitudes (co-operative) Self-InterestValues & attitudes (Individualistic) Community OrientationValues & attitudes (co-operative) KnowledgeKnowledge & Ability Negative Peer PressureRelated to competitive value? Positive Peer PressureRelated to competitive value? OpportunitiesInstitutional factors

13. Next Steps… Adapting and finalising scenarios to test for priority values, attitudes and critical thinking skills Adapting and finalising scenarios to test for priority values, attitudes and critical thinking skills Piloting tool in DE/ESD sector Piloting tool in DE/ESD sector Analysing data Analysing data Reporting data obtained to sector and Reporting data obtained to sector and fine-tuning tool fine-tuning tool

14. Thank-you RCE-IRELAND Nancy L. Serrano, RCE Research Assistant, Department of Education and Professional Studies, University of Limerick Phone: 061-212927 Phone: 061-212927 E-mail: nancy.l.serrano@ul.ie