1. The Development of Rubrics to Measure Undergraduate Students’ Global Awareness and Global Perspective: A Validity Study Stephanie Paul Doscher Department of Leadership and Professional Studies May 25 th, 2011

2. Study Meets Many Needs Researcher Institution Higher Education

3. Purpose of the Study Demonstrate the extent to which evidence supports the validity and reliability of scores yielded from rubrics designed to measure undergraduate students’ global awareness and global perspective

4. Cognitive Flexibility Theory Rand Spiro, Michigan State University Ill-structured vs. well-structured domains Knowledge transfer to unique cases Context-dependent protocols

5. Assumptions Global learning strategies are substantively different from traditional higher education teaching strategies Global learning strategies enable students to develop global awareness and global perspective

6. Research Questions 1.To what extent does evidence support the reliability of scores yielded from a rubric measuring students’ global awareness? 2.To what extent does evidence support the reliability of scores yielded from a rubric measuring students’ global perspective?

7. Research Questions 3. To what extent does evidence support the validity of scores yielded from a rubric measuring students’ global awareness? 4. To what extent does evidence support the validity of scores yielded from a rubric measuring students’ global perspective?

8. Hypotheses Research questions 1 & 2: one hypothesis –Inter-rater agreement

9. Hypotheses Research questions 3 & 4: six hypotheses –Class status –Race/ethnicity –Fluency in more than one language –Time spent abroad –Previous global learning course completion –Global learning course grade

10. Research Design PretestTreatmentPosttest Experimental groupO 1 X 1 O 1 - - - - - - - - - - - - - - - Control groupO 2 X 2 O 2

11. Sample Experimental group: students enrolled in two different global learning courses in separate sections of the University Core Curriculum (UCC) Control group: students enrolled in two separate non-global learning courses in same UCC sections as participating global learning courses

12. Instrumentation Parallel case studies –“The Problem of Hoodia” –“A Monumental Dilemma Questions –What is the problem/dilemma in this case? (global awareness) –What perspectives need to be taken into account in order to find a solution to this problem/dilemma? (global perspective) Rubrics

13. Data Collection Emails to chairs and potential faculty Pretest—within first two weeks of class Posttest—within last two weeks of class Trained faculty raters Two raters score each question, third rater for discrepant scores

14. Data Analysis Inter-rater agreement Multiple linear regression (ANCOVA) to test hypotheses –One-tailed test of significance –Within group analysis