The Institutional Artifact Portfolio Process: An Effective and Nonintrusive Method for General Education Assessment The Institutional Artifact Portfolio Process: An Effective and Nonintrusive Method for General Education Assessment Renée M. Tobin, Derek J. Herrmann, & Kelly L. Whalen Illinois State University Renée M. Tobin, Derek J. Herrmann, & Kelly L. Whalen Illinois State University

Illinois State University: Who We Are DRU Fall 2010 census day figures Total enrollment: 20,762 Undergraduate enrollment: 18, % of all students are female 13.1% of all students are minority

Illinois State University: Who We Are 96.8% of undergrads are from Illinois 93.1% of undergrads are < 24 years old 94.6% of undergrads are full-time students

Illinois State University: Who We Are Fact Book (Planning and Institutional Research) – Fall 2010 First-year students ACT mean composite scores: – Illinois State 24.1 – State of Illinois20.7 – National21.0

Illinois State University: Who We Are Transfer Students – 1,838 new transfer students in Fall 2010 – ~50% of bachelor’s degrees granted annually 85.0% retention from first-year to sophomore

University Assessment Services Illinois State University UAS Office of the Provost IRFinance and Planning The two offices work closely; however, UAS is responsible for the management of all program-level assessment.

History of ISU’s Gen Ed Program Senate approval in 1992 Full implementation in 1998 Assessment task force in 2005 Full implementation in 2008 New task force in – Assessment subcommittee!

Current Gen Ed Program 12 Goals with 40 distinct skills/abilities 42 credit hours 190 potential courses Approximately 13,410 enrollments per semester 3 Cores: – Inner – (5 courses) – Middle – (5 courses) – Outer – (4 courses)

Current Gen Ed Assessment Comprehensive, yet manageable Purpose: To provide the Council for General Education with sound evidence to base decisions regarding the program.

Important Contributors AKA Campus Buy-In Director of General Education Council for General Education General Education Assessment Task Force Assessment Advisory Council

Research Reviewed past methodologies employed at ISU Reviewed methodologies employed at other institutions

Initial Challenge Goals of General Education – Numerous (12 with 40 abilities) – Difficult for faculty/staff/students to remember and reference – In some cases, challenging to measure

Solution Four Shared Learning Outcomes Common and integrated elements of the established goals of Gen Ed Also aimed at eliminating some of the division that is present between Gen Ed and the Major

Selecting an Assessment Method: The Institutional Portfolio (Seybert) Principles – Gen Ed is responsibility of entire campus – Be minimally intrusive (both faculty and students) – Use existing student work

Selecting an Assessment Method: The Institutional Portfolio (Seybert) Process – “Artifacts” are collected – Faculty teams review using rubrics – Results are compiled and reported to committee – Committee makes decisions based on results

Selecting an Assessment Method: The Institutional Portfolio (Seybert) Characteristics – Faculty review teams are multidisciplinary – Review is invisible to students and not intrusive to faculty – Process is labor intensive and requires resources – Process is dynamic and “messy”

Method – Requirements

Non-intrusive Cover the 12 goals of general education Comprehensive, Manageable Institution-focused

IAP Schedule Fall 2008Public Opportunity Spring 2009Critical Inquiry and Problem Solving Fall 2009Diverse and Global Perspectives Spring 2010Life-Long Learning Fall 2010Critical Inquiry and Problem Solving Spring 2011Public Opportunity Fall 2011Life-Long Learning Spring 2012Diverse and Global Perspectives

General Overview for Assessment Phase One: Obtaining the artifacts Phase Two: Sampling the artifacts Phase Three: Reviewing the artifacts

Phase One: Obtaining the Artifacts Instructor Participation: 1.Read the invitation letter from Provost 2.Identify artifact(s) that address at least 1 Primary Trait 3.Complete the online participation form 4.Allow UAS access to the artifacts for up to 24 hours

Shared Learning Outcome – Public Opportunity (Students will identify the resources and articulate the subsequent value of civic and community engagement.) Primary Traits for Public Opportunity: 1.Critically informed position on civic life 2.Influence of civic participation on the social and collaborative nature of knowledge 3.Contributions to the public affecting individual life aspects- such as family, religion, business and/or the state 4.Contributions to the public affecting social and community life aspects – such as family, religion, business and/or the state 5.Resources for civic engagement 6.Civic participation in the social, economic, technological, and/or political dimensions of community development 7.Self-Reflection (Elective) 8.Discipline Knowledge (Elective)

Phase Two: Sampling the Artifacts Goal – Obtain a random sample of 100 artifacts per core (300 total artifacts per Shared Learning Outcome) Behind the Scenes at UAS: Calculate the total number of enrollments, based on class size Remove all identifying information and oversample End of the term, recalculate and use the appropriate proportions

Sampling Example Course Dept. & NumberInstructorCoreSection # # of artifacts # of students% of total # needed for final sample # to sample and copy ABC 200AOuter BCD 100BOuter BCD 100COuter BCD 100COuter DEF 200DOuter BCD 200EOuter BCD 200FOuter EFG 100GOuter EFG 200HOuter EFG 200GOuter EFG 200GOuter DEF 200IOuter DEF 200JOuter FGH 200KOuter BCD 100LOuter TOTAL

Sampling Example Course Dept. & NumberInstructorCoreSection # # of artifacts # of students% of total # needed for final sample # to sample and copy ABC 200AOuter BCD 100BOuter BCD 100COuter BCD 100COuter DEF 200DOuter BCD 200EOuter BCD 200FOuter EFG 100GOuter EFG 200HOuter EFG 200GOuter EFG 200GOuter DEF 200IOuter DEF 200JOuter FGH 200KOuter BCD 100LOuter TOTAL

Sampling Example Course Dept. & NumberInstructorCoreSection # # of artifacts # of students% of total # needed for final sample # to sample and copy ABC 200AOuter BCD 100BOuter BCD 100COuter BCD 100COuter DEF 200DOuter BCD 200EOuter BCD 200FOuter EFG 100GOuter EFG 200HOuter EFG 200GOuter EFG 200GOuter DEF 200IOuter DEF 200JOuter FGH 200KOuter BCD 100LOuter TOTAL

Phase Three: Reviewing the Artifacts 1.Instructors solicited to apply to be IAP reviewers 2.Interdisciplinary two-person teams (3 per Shared Learning Outcome and an alternate) are trained in use of established rubrics 3.Review teams complete consensus analysis using developed rubrics for the 4 Shared Learning Outcomes

Review Process Review Week: Day One: – Reviewers interpret the rubrics as a group – Calibration training using practice artifacts Days Two through Four: – Reviewers split into teams and complete their binders – Final rubrics collected throughout

Phase Three: Reviewing the Artifacts

Excerpt from the Public Opportunity Rubric: Primary Traits Not PresentDevelopingEstablishedAdvanced Gen Ed Goal Critically informed position on civic life Describes the value of contributions to civic life in the dimensions of their own life Compares the value of contributions to civic life from multiple, critically informed perspectives Defends or refutes the value of contributions to civic life 7.c Influence of civic participation on the social and collaborative nature of knowledge Identifies how civic participation can change the social and collaborative nature of knowledge Explains how civic participation can change the social and collaborative nature of knowledge Applies new knowledge in the context of civic participation 12

Phase Three: Reviewing the Artifacts 1.Instructors solicited to apply to be IAP reviewers 2.Interdisciplinary two-person teams (3 per Shared Learning Outcome and an alternate) are trained in use of established rubrics 3.Review teams complete consensus analysis using developed rubrics for the 4 Shared Learning Outcomes 4.The final data are reported to Council of General Education to formulate commendations/recommendations

Results

Reliability – Inter-rater reliability (calibration artifacts) Percent agreement – Public Opportunity: 42% - 71% – Critical Inquiry and Problem Solving: 61% - 69% Intraclass correlation coefficient – Diverse and Global Perspectives: – Life-Long Learning: – Test-retest reliability (repeat calibration artifacts) Exploratory factor analysis

Results Focus on trends for each Shared Learning Outcome No consistent differences across freshmen, sophomores, juniors, and seniors No clear differences emerged when comparing native and transfer students

Results Maximum review value consistent with instructions

Discussion Limitations of the results – Work in progress – Expand from 3 Cores to many course categories – Data based on courses from one semester only

Discussion Future directions – More time calibrating during review process – Reports with data from two different semesters – General Education Task Force (and its Assessment subcommittee)

Conclusions Advantages of the IAP – Nonintrusive to instructional faculty and staff and students – Allows for program-level assessment – Maintains the institution as the focus

Conclusions Challenges of the IAP – Faculty participation – Locus of generalizability – Takes time

Conclusions Lessons learned – Patience and public relations – Streamline the process – Oversample – Reviewer calibration is key – Alternate reviewer – Closing the loop

QUESTIONS?

For More Information

Illinois State University University Assessment Services Normal, IL Renée M. Tobin, Ph.D. Derek J. HerrmannKelly L. Whalen Acting Director Coordinator Graduate Assistant