Presentation is loading. Please wait.

Presentation is loading. Please wait.

Graduate Program Assessment: A Pilot Study Using a Common Activity and Combined Rubric Rana Khan, Ph.D., Director, Biotechnology Program Datta Kaur Khalsa,

Published byBrendan Willis Modified over 9 years ago

Similar presentations

Presentation on theme: "Graduate Program Assessment: A Pilot Study Using a Common Activity and Combined Rubric Rana Khan, Ph.D., Director, Biotechnology Program Datta Kaur Khalsa,"— Presentation transcript:

1 Graduate Program Assessment: A Pilot Study Using a Common Activity and Combined Rubric Rana Khan, Ph.D., Director, Biotechnology Program Datta Kaur Khalsa, Ph.D., Director of Assessment, Education Department Kathryn Klose, Ph.D., Associate Chair & Director, Finance Management and Accounting Yan Cooksey, Ph.D. Director, Learning Outcomes Assessment, Dean’s Office Sloan Conference Oct 11, 2012

2 Course Outcomes Program Objectives Undergraduate and Graduate School Goals Institutional Outcomes UMUC’s LEVELS OF ASSESSMENT Sloan Conference Oct 11, 2012

3 STUDENT LEARNING EXPECTATIONS (SLEs) Written Communication (COMM) Produce writing that meets expectations for format, organization, content, purpose, and audience. Information Literacy (INFO) Demonstrate the ability to use libraries and other information resources to effectively locate, select, and evaluate needed information. Critical Thinking (THIN) Demonstrate the use of analytical skills and reflective processing of information. Technology Fluency (TECH) Demonstrate an understanding of information technology broad enough to apply technology productively to academic studies, work, and everyday life. Content/Discipline-Specific Knowledge (KNOW) Demonstrate knowledge and competencies specific to program or major area of study. UMUC GRADUATE SCHOOL SLEs Sloan Conference Oct 11, 2012

4 CURRENT APPROACH: 3-3-3 MODEL 3 rounds, over 3 years, at 3 stages 5 SLEs: COMM, THIN, INFO, TECH, KNOW Sloan Conference Oct 11, 2012

5 3-3-3 Model ASSESSING THE ASSESSMENT Strengths:Weaknesses: Tested rubricsAdded faculty workload Reasonable collection points Lack of consistency in assignments Larger samples - more data for analysis Variability in applying scoring rubrics Sloan Conference Oct 11, 2012

6 Common activity Topic for all disciplines – “Challenges facing leaders” Combined activity 4 SLEs (all except KNOW) SLE criteria from existing rubrics – eliminate overlap 4-pt scale (Exemplary, Competent, Marginal & Unsatisfactory) Training raters and norming COMBINED ACTIVITY/RUBRIC (C2) MODEL Sloan Conference Oct 11, 2012

7 Current 3-3-3 Model Combined Activity/Rubric (C2) Model Multiple Rubrics: one for each of 4 SLEs Single rubric for all 4 SLEs Multiple assignments across graduate school Single assignment across graduate school One to multiple courses/4 SLEsSingle course/4 SLEs Multiple raters for the same assignment/course Same raters/assignment/course Untrained ratersTrained raters 3-3-3 VS COMBINED ACTIVITY/RUBRIC (C2) MODEL Sloan Conference Oct 11, 2012

8 DESIGN OF A PILOT STUDY Purpose: –To simplify the current assessment process –To increase the process reliability and validity Methods: –Courses were identified –Faculty chosen to be raters –Norming sessions were conducted –Paper were collected and assessed –Intra-Class Correlation Coefficient (ICC) was calculated Sloan Conference Oct 11, 2012

9 Implementation Process of Pilot Study Week 1 Norming Session 1: Rater orientation of scoring process, activity, rubric and timeline Week 2 Scoring Session 1: Anchor paper grading Norming Session 2: Asynchronous comparative discussion Week 3 Norming Session 3: Live conference discussing anchor results and rubric questions Week 4 Scoring Session 2: 10-day grading period of all student papers by raters Week 5 Norming Session 4: Live conference on results with feedback for improvement Week 6 Pilots student data processed and analyzed SPRING 2012 PILOT NORMING Sloan Conference Oct 11, 2012

10 Intra-class Correlation Coefficient (ICC) estimation of inter-rater reliability one-way random effects ANOVA model PHASE I PILOT RESULTS >0.75, excellent; 0.40 to 0.75, fair to good/moderate; <0.40 poor Source: Fleiss (1986) on ICC values clinical & social science research Sloan Conference Oct 11, 2012

11 Item Intra-class Correlation THIN Conceptualization0.396 Analysis0.493 Synthesis0.509 Conclusion0.390 Implications0.201 INFO Evaluation0.430 Incorporation0.381 Ethical Use0.335 COMM Context/Purpose0.316 Content/Ideas/Support0.475 Organization0.444 Grammar/Spelling/Punctuation0.456 TECH Tech Mgmt.0.175 Info Retrieval0.512 PHASE I PILOT RESULTS Sloan Conference Oct 11, 2012

12 PHASE I PILOT RESULTS Overall Descriptive Statistics (N=91) MeanMedianMode Std. Deviation MinimumMaximum THIN Conceptualization3.2473.3503.5.48231.84.0 Analysis3.0913.1503.3.51941.54.0 Synthesis3.0533.1503.5.58391.54.0 Conclusion2.9983.1003.5.60241.14.0 Implication2.9353.0003.0.60261.04.0 INFO Evaluation3.2583.2503.3.49612.04.0 Incorporation3.1583.2503.3.51911.54.0 Ethical Use3.5793.7504.0.54251.04.0 COMM Context/Purpose3.2253.3003.5.49001.84.0 Content/Ideas/Support3.1053.1503.3.50792.04.0 Organization3.1163.2503.3.55751.54.0 Grammar/Spelling/ Punctuation 3.1153.2503.5.55931.84.0 TECH Tech Mgmt.3.6213.7503.8.42131.84.0 Info Retrieval3.6393.7503.8.41101.04.0 Total45.09345.80045.2 a 5.554129.555.4 a.Multiple modes exist. The smallest value is shown b.Scale: Exemplary: 3.1-4.0; Competent: 2.1-3.0; Marginal 1.1-2.0; Unsatisfactory: 0-1.0 Sloan Conference Oct 11, 2012

13 PILOT INTENTIONS  Consistency in interpretation of rubric  Consistency in use of rubric  Address variability of data collection  Limit extra load on faculty Sloan Conference Oct 11, 2012

14 LESSONS LEARNED Review alignment Consolidate rubric further Tech management criteria Norming practice Sloan Conference Oct 11, 2012

15 “Refined Rubric and Random Paper Grading Study.” Same raters Same papers but distributed randomly More norming practice with the refined rubric Increase evidence of combined rubric validity FUTURE DIRECTION - PHASE II Sloan Conference Oct 11, 2012

16

17 REFERENCE Fleiss, J. L. (1986). Design and analysis of clinical experiments. New York, NY: John Wiley & Sons. Sloan Conference Oct 11, 2012

18 CONTACT Rana Khan : rana.khan@umuc.edu rana.khan@umuc.edu Datta Kaur-Khalsa : dattakaur.khalsa@umuc.edu dattakaur.khalsa@umuc.edu Kathryn Klose : kathryn.klose@umuc.edu kathryn.klose@umuc.edu Yan Cooksey : yan.cooksey@umuc.eduyan.cooksey@umuc.edu Sloan Conference Oct 11, 2012

19 ACKNOWLEDGEMENTS John Aje Diane Bartoo Nancy Glenn Kathy Marconi Dan McCollum Garth McKenzie Pat Spencer Rudy Watson Bruce Katz Dawn Rodriguez Carol De’Arment Pat Miller Lisa Parsons Katie Crockett Anthony Cristillo Sloan Conference Oct 11, 2012

Download ppt "Graduate Program Assessment: A Pilot Study Using a Common Activity and Combined Rubric Rana Khan, Ph.D., Director, Biotechnology Program Datta Kaur Khalsa,"

Similar presentations

Global Learning Outcomes at Pensacola State College (GLOs)

Global Learning Outcomes at Pensacola State College (GLOs)
Pathways for Assessing Scientific Literacy, Fluency in Technology, and Information Literacy Kathleen Warner, PhD Janet Zimmer, PhD Elizabeth Mulherrin,

Pathways for Assessing Scientific Literacy, Fluency in Technology, and Information Literacy Kathleen Warner, PhD Janet Zimmer, PhD Elizabeth Mulherrin,
The Teacher Work Sample

The Teacher Work Sample
Daniel Peck January 28, 2011. SLOs versus Course Objectives Student Learning Outcomes for the classroom describe the knowledge, skills, abilities.

Daniel Peck January 28, SLOs versus Course Objectives Student Learning Outcomes for the classroom describe the knowledge, skills, abilities.
Carol Ann Gittens, Gail Gradowski & Christa Bailey Santa Clara University WASC Academic Resource Conference Session D1 April 25, 2014.

Carol Ann Gittens, Gail Gradowski & Christa Bailey Santa Clara University WASC Academic Resource Conference Session D1 April 25, 2014.
STUDENT LEARNING OUTCOMES ASSESSMENT. Cycle of Assessment Course Goals/ Intended Outcomes Means Of Assessment And Criteria For Success Summary of Data.

STUDENT LEARNING OUTCOMES ASSESSMENT. Cycle of Assessment Course Goals/ Intended Outcomes Means Of Assessment And Criteria For Success Summary of Data.
Working with Rubrics: Using the Oral Communication, Writing, and Critical Thinking Rubrics VALUE Rubrics Ashley Finley, Ph.D Senior Director of Assessment.

Working with Rubrics: Using the Oral Communication, Writing, and Critical Thinking Rubrics VALUE Rubrics Ashley Finley, Ph.D Senior Director of Assessment.
Issues of Technical Adequacy in Measuring Student Growth for Educator Effectiveness Stanley Rabinowitz, Ph.D. Director, Assessment & Standards Development.

Issues of Technical Adequacy in Measuring Student Growth for Educator Effectiveness Stanley Rabinowitz, Ph.D. Director, Assessment & Standards Development.
22 May 2012. Session Overview Purposes: Provide an overview of the assessment plan for Foundational Studies Discuss Phase I of the assessment plan – assessing.

22 May Session Overview Purposes: Provide an overview of the assessment plan for Foundational Studies Discuss Phase I of the assessment plan – assessing.
OUTCOMES ASSESSMENT Developing and Implementing an Effective Plan.

OUTCOMES ASSESSMENT Developing and Implementing an Effective Plan.
Susan Malone Mercer University.  “The unit has taken effective steps to eliminate bias in assessments and is working to establish the fairness, accuracy,

Susan Malone Mercer University.  “The unit has taken effective steps to eliminate bias in assessments and is working to establish the fairness, accuracy,
Crowd-Sourcing Innovative Practices: Assessing Integrative Learning at Large Research Institutions.

Crowd-Sourcing Innovative Practices: Assessing Integrative Learning at Large Research Institutions.
Measuring Student Learning March 10, 2015 Cathy Sanders Director of Assessment.

Measuring Student Learning March 10, 2015 Cathy Sanders Director of Assessment.
Educational Outcomes: The Role of Competencies and The Importance of Assessment.

Educational Outcomes: The Role of Competencies and The Importance of Assessment.
Hold the door: Crossing the threshold from ACRL Information Literacy Standards to the new Framework OCULA Spring Conference 2015 Nancy Birch & Christopher.

Hold the door: Crossing the threshold from ACRL Information Literacy Standards to the new Framework OCULA Spring Conference 2015 Nancy Birch & Christopher.
What’s in the works for General Education Assessment? Dan McCollum, Ph.D. Associate Director of Academic Assessment.

What’s in the works for General Education Assessment? Dan McCollum, Ph.D. Associate Director of Academic Assessment.
Developing General Education Course Assessment Measures Anthony R. Napoli, PhD Lanette A. Raymond, MA Office of Institutional Research & Assessment Suffolk.

Developing General Education Course Assessment Measures Anthony R. Napoli, PhD Lanette A. Raymond, MA Office of Institutional Research & Assessment Suffolk.
Diana Chan, HKUST Enhancing Learning Experiences in Higher Education: International Conference 2 -3 December 2010.

Diana Chan, HKUST Enhancing Learning Experiences in Higher Education: International Conference 2 -3 December 2010.
Office of Research, Evaluation, and Assessment April 19, 2008.

Office of Research, Evaluation, and Assessment April 19, 2008.
EPortfolio Assessment Pilot. Agenda Purpose of the ePortfolio assessment pilot CSD use of ePortfolio English department use of ePortfolio Future applications.

EPortfolio Assessment Pilot. Agenda Purpose of the ePortfolio assessment pilot CSD use of ePortfolio English department use of ePortfolio Future applications.

Similar presentations

Ads by Google