Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education and Assessment Conference Boston, Massachusetts, March 1 st, 2013
Session Outline Global FIU Rubric anatomy Developing the rubrics Scorer training Rubric for rubrics Gathering evidence of validity and reliability
Text
FIUs Global Learning SLOs Global Awareness: knowledge of the interrelatedness of local, global, international and intercultural issues, trends, and systems Global Perspective: ability to develop a multi- perspective analysis of local, global, international, and intercultural problems Global Engagement: willingness to engage in local, global, international, and intercultural problem solving
Active Learning Strategies/ Performance Assessments Essay, story, or poem Research paper Literary analysis Book or article review Case study Speech Journal response Art exhibit Portfolio Musical composition Lab report Strategic planning In-class discussion Editorials Peer editing Poster presentation Video Podcast Mock trial Oral presentation Debate Role play Online discussion Blogs Advertising campaign Building a prototype Modeling Experiments Service learning
Developing the Rubrics Faculty learning community Pilot case studies, questions, scoring criteria Revisions based on response trends Pilot study Benchmark responses Anchor papers Faculty feedback, expert judge feedback Two field tests Scorer training Rubric language, response minimum
Scorer Training Full-time and adjunct faculty Pre-training packet Open discussion and review of cases, questions, and rubrics Norming session with anchor papers Sample scoring session, 10% of total papers
Validating the Rubrics Validity: Can the rubrics detect the differences in students development of global awareness and global perspective? Reliability: Can trained faculty raters agree on rubric scores 80% of the time or more?
Validating the Rubrics PretestTreatmentPosttest Global learning O 1 X 1 O Non-global learningO 2 X 2 O 2 Research Design
Data Collection s to chairs and potential faculty Pretestwithin first two weeks of class Posttestwithin last two weeks of class Trained faculty raters Two raters scored each question, third rater for discrepant scores
Results Inter-rater reliability Global awareness rubric Pretest (.89, p <.0001) Posttest (.95, p <.0001) Global perspective rubric Pretest (.92, p <.0001) Posttest (.91, p <.0001)
Results Validity – Global Awareness Rubric No significant main effects Post-hoc analysis: Groups differed significantly on pretest scores, p =.003 Global learning (M = 1.51) Non-global learning (M = 1.85) Significant interaction between global awareness pretest score and the treatment in predicting global awareness posttest score, p =.005
Global Awareness Rubric
Results Validity – Global Perspective Rubric No significant main effects Post-hoc analysis: Groups differed significantly on pretest scores, p =.003 Global learning (M =.90) Non-global learning (M = 1.2) Significant interaction between global perspective pretest score and the treatment in predicting global perspective posttest score, p =.005
Results
Interpretation of Results Reliability Both rubrics highly reliable Consistent with literature on rubric development and rater training Empirical support for structural validity
Interpretation of Results Validity Rubric detects differences between global learning and non-global learning students Rubric detects differences within the group of global learning students
Thank You! For more information, please contact me: Stephanie Doscher, Office of Global Learning Visit our web site: goglobal.fiu.edu