The Development of Rubrics to Measure Undergraduate Students’ Global Awareness and Global Perspective: A Validity Study Stephanie Paul Doscher Department.

Slides:



Advertisements
Similar presentations
Developing Valid Rubrics for Assessing Global Awareness and Global Perspective Dr. Stephanie Doscher Florida International University AAC&U General Education.
Advertisements

Strategies methods strengths challenges What did Lane learn?
Making Your Assessments More Meaningful Flex Day 2015.
1 ICT-based Paradigm Shifts Dave Moursund Teacher Education.
 I would like to thank my dear teachers who have taken all the trouble to come to this remote place in upper Egypt.
4/16/07 Assessment of the Core – Social Inquiry Charlyne L. Walker Director of Educational Research and Evaluation, Arts and Sciences.
Educational Outcomes: The Role of Competencies and The Importance of Assessment.
International Conference on Lifelong Leaning ICLLL 2011
Research Proposal Development of research question
Effective Grading and Assessment:. Strategies to Enhance Student Learning.
FLCC knows a lot about assessment – J will send examples
Department of Humanities College of Sciences and Liberal Arts Writing Program Assessment at New Jersey Institute of Technology Carol Siri Johnson Associate.
Ismaila Odogba Department of Geography and Geology The Impact of Collaborative Testing on Student Learning in an Introductory Level Geography Course INTRODUCTION.
PARCC Information Meeting FEB. 27, I Choose C – Why We Need Common Core and PARCC.
 Description  The unit has a conceptual framework that defines how our programs prepare candidates to be well-rounded educators. Every course in the.
Broome-Tioga BOCES Network Team… the ENTIRE team! (Pat Walsh, Barb Phillips, Jennifer Dove)
Evidence-Based Practice Current knowledge and practice must be based on evidence of efficacy rather than intuition, tradition, or past practice. The importance.
Updating the National Survey of Student Engagement: Analyses of the NSSE 2.0 Pilots Allison BrckaLorenz Bob Gonyea Angie Miller.
Measuring Changes in Teachers’ Mathematics Content Knowledge Dr. Amy Germuth Compass Consulting Group, LLC.
Assessment for Optimal Learning Tace Crouse Faculty Center for Teaching and Learning University of Central Florida.
Measuring Changes in Teachers’ Science Content Knowledge Dr. Anne D’Agostino Compass Consulting Group, LLC.
Everything you wanted to know about Assessment… Dr. Joanne Coté-Bonanno Barbara Ritola September 2009 but were afraid to ask!
Department of English & Foreign Languages B.A. in English CIP Code: Program Code: Program Quality Improvement Report
NAVIGATING THE PROCESS OF STUDENT LEARNING OUTCOMES: DEVELOPMENT, EVALUATION, AND IMPROVEMENT Shannon M. Sexton, Julia M. Williams, & Timothy Chow Rose-Hulman.
University of Arkansas Faculty Senate Task Force on Grades Preliminary Report April 19, 2005.
S oTL Project Wisconsin Teaching Scholars Program, Dr. David M. Jones Department of English/Women’s Studies Affiliate EDI/Honors Fellow, University.
Assessing General Education Workshop for College of the Redwoods Fred Trapp August 18, 2008.
Department of English and Foreign Languages School of Liberal Arts B.A. in English Education CIP Code Program Code Program Quality Improvement.
Assessment of Student Presentations Jones College of Business MTSU July 8, 2015.
Differential Effects of Participatory Evaluation in a National Multi-site Program Evaluation Frances Lawrenz University of Minnesota.
Invention Convention Seth Krivohlavek Angie Deck.
Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research.
María Pita Carranza Ángel Centeno Ángela Corengia Laura Llull Belén Mesurado Cecilia Primogerio Francisco Redelico Predicting Academic Performance and.
Reliability vs. Validity.  Reliability  the consistency of your measurement, or the degree to which an instrument measures the same way each time it.
A Comparative Study of Online Discussion Board Protocols and the Impact on Knowledge Construction in High School Students Cara Kugler CECS 5610.
Dr. Roxana Reichman Gordon Academic College Haifa, ISRAEL.
Connecting Course Goals, Assignments, and Assessment Faculty Development for Student Success at Prince George’s Community College William Peirce
Competency Assessment Advisory Team (CAAT) QUANTITATIVE REASONING DEPARTMENT OF MATHEMATICS REP – ROB NICHOLS 1.
VALUE/Multi-State Collaborative (MSC) to Advance Learning Outcomes Assessment Pilot Year Study Findings and Summary These slides summarize results from.
Examining Rubric Design and Inter-rater Reliability: a Fun Grading Project Presented at the Third Annual Association for the Assessment of Learning in.
QLP Experiences: CRJ 3013 Drs. Rob Tillyer & Megan Augustyn Department of Criminal Justice Presentation to the QLP Group August 21, 2015.
Evaluating Impacts of MSP Grants Ellen Bobronnikov January 6, 2009 Common Issues and Potential Solutions.
Evidence-Based Practice Evidence-Based Practice Current knowledge and practice must be based on evidence of efficacy rather than intuition, tradition,
PDE Literacy Design Collaborative Grant February 16, 2011.
Minnesota's Approach to Comprehensive Assessment Megan E. Cox, Ph.D. Principal Leadership Academy January 11, 2016 Minnesota’s Approach to Comprehensive.
1 Learning Outcomes Assessment: An Overview of the Process at Texas State Beth Wuest Director, Academic Development and Assessment Lisa Garza Director,
Module 3: Programmatic Assessment Strategies
Teacher Incentive Fund U.S. Department of Education.
MT ENGAGE Student Learning Outcomes and Assessment April 27, 2015.
Preparing for the Title III Part F STEM Competition Alliance of Hispanic Serving Institutions Educators Grantsmanship Institute March 20, 2016.
Encouraging Faculty Development Where You Can: Using a Review of Assessment Management Software Joanne Liebman Matson | Belinda Blevins-Knabe University.
Overview of Types of Measures Margaret Kasimatis, PhD VP for Academic Planning & Effectiveness.
Reviewing Syllabi to Document Teaching Culture and Inform Decisions Claudia J. Stanny Director, Center for University Teaching, Learning, & Assessment.
Talk about the assignment! April 27th 2015 #TOOC15 Webinar.
CAEP Standard 4 Program Impact Case Study
Evaluation Requirements for MSP and Characteristics of Designs to Estimate Impacts with Confidence Ellen Bobronnikov March 23, 2011.
Assessment Fall Quarter 2007.
Research Study.
Comparison of Nurse Mentor and Instructor
Surface energy modification for biomedical material by corona streamer plasma processing to mitigate bacterial adhesion Ibrahim Al-Hamarneh Patrick.
Helping students know what they know
Outcome Assessment Using a Total Quality Management Paradigm
Measuring Teaching Practices
Jennifer Bryer PhD, RN, CNE Virginia Peterson-Graziose DNP, RN, CNE
Evaluating Online Courses and the Challenges Associated
Global Awareness Institutional Learning Outcome Assessment
Kelley Morel, M. D. , Fidel Valea, M. D. , Allison Tegge, Ph. D
Developing a Rubric for Assessment
Business Administration Programs School of Business and Liberal Arts Fall 2016 Assessment Report
Assessment Fall Quarter 2007.
Presentation transcript:

The Development of Rubrics to Measure Undergraduate Students’ Global Awareness and Global Perspective: A Validity Study Stephanie Paul Doscher Department of Leadership and Professional Studies May 25 th, 2011

Study Meets Many Needs Researcher Institution Higher Education

Purpose of the Study Demonstrate the extent to which evidence supports the validity and reliability of scores yielded from rubrics designed to measure undergraduate students’ global awareness and global perspective

Cognitive Flexibility Theory Rand Spiro, Michigan State University Ill-structured vs. well-structured domains Knowledge transfer to unique cases Context-dependent protocols

Assumptions Global learning strategies are substantively different from traditional higher education teaching strategies Global learning strategies enable students to develop global awareness and global perspective

Research Questions 1.To what extent does evidence support the reliability of scores yielded from a rubric measuring students’ global awareness? 2.To what extent does evidence support the reliability of scores yielded from a rubric measuring students’ global perspective?

Research Questions 3. To what extent does evidence support the validity of scores yielded from a rubric measuring students’ global awareness? 4. To what extent does evidence support the validity of scores yielded from a rubric measuring students’ global perspective?

Hypotheses Research questions 1 & 2: one hypothesis –Inter-rater agreement

Hypotheses Research questions 3 & 4: six hypotheses –Class status –Race/ethnicity –Fluency in more than one language –Time spent abroad –Previous global learning course completion –Global learning course grade

Research Design PretestTreatmentPosttest Experimental groupO 1 X 1 O Control groupO 2 X 2 O 2

Sample Experimental group: students enrolled in two different global learning courses in separate sections of the University Core Curriculum (UCC) Control group: students enrolled in two separate non-global learning courses in same UCC sections as participating global learning courses

Instrumentation Parallel case studies –“The Problem of Hoodia” –“A Monumental Dilemma Questions –What is the problem/dilemma in this case? (global awareness) –What perspectives need to be taken into account in order to find a solution to this problem/dilemma? (global perspective) Rubrics

Data Collection s to chairs and potential faculty Pretest—within first two weeks of class Posttest—within last two weeks of class Trained faculty raters Two raters score each question, third rater for discrepant scores

Data Analysis Inter-rater agreement Multiple linear regression (ANCOVA) to test hypotheses –One-tailed test of significance –Within group analysis