The Development of Rubrics to Measure Undergraduate Students’ Global Awareness and Global Perspective: A Validity Study Stephanie Paul Doscher Department of Leadership and Professional Studies May 25 th, 2011
Study Meets Many Needs Researcher Institution Higher Education
Purpose of the Study Demonstrate the extent to which evidence supports the validity and reliability of scores yielded from rubrics designed to measure undergraduate students’ global awareness and global perspective
Cognitive Flexibility Theory Rand Spiro, Michigan State University Ill-structured vs. well-structured domains Knowledge transfer to unique cases Context-dependent protocols
Assumptions Global learning strategies are substantively different from traditional higher education teaching strategies Global learning strategies enable students to develop global awareness and global perspective
Research Questions 1.To what extent does evidence support the reliability of scores yielded from a rubric measuring students’ global awareness? 2.To what extent does evidence support the reliability of scores yielded from a rubric measuring students’ global perspective?
Research Questions 3. To what extent does evidence support the validity of scores yielded from a rubric measuring students’ global awareness? 4. To what extent does evidence support the validity of scores yielded from a rubric measuring students’ global perspective?
Hypotheses Research questions 1 & 2: one hypothesis –Inter-rater agreement
Hypotheses Research questions 3 & 4: six hypotheses –Class status –Race/ethnicity –Fluency in more than one language –Time spent abroad –Previous global learning course completion –Global learning course grade
Research Design PretestTreatmentPosttest Experimental groupO 1 X 1 O Control groupO 2 X 2 O 2
Sample Experimental group: students enrolled in two different global learning courses in separate sections of the University Core Curriculum (UCC) Control group: students enrolled in two separate non-global learning courses in same UCC sections as participating global learning courses
Instrumentation Parallel case studies –“The Problem of Hoodia” –“A Monumental Dilemma Questions –What is the problem/dilemma in this case? (global awareness) –What perspectives need to be taken into account in order to find a solution to this problem/dilemma? (global perspective) Rubrics
Data Collection s to chairs and potential faculty Pretest—within first two weeks of class Posttest—within last two weeks of class Trained faculty raters Two raters score each question, third rater for discrepant scores
Data Analysis Inter-rater agreement Multiple linear regression (ANCOVA) to test hypotheses –One-tailed test of significance –Within group analysis