2. The Institutional Artifact Portfolio Process: An Effective and Nonintrusive Method for General Education Assessment The Institutional Artifact Portfolio Process: An Effective and Nonintrusive Method for General Education Assessment Renée M. Tobin, Derek J. Herrmann, & Kelly L. Whalen Illinois State University Renée M. Tobin, Derek J. Herrmann, & Kelly L. Whalen Illinois State University

3. Illinois State University: Who We Are DRU Fall 2010 census day figures Total enrollment: 20,762 Undergraduate enrollment: 18,254 64.4% of all students are female 13.1% of all students are minority

4. Illinois State University: Who We Are 96.8% of undergrads are from Illinois 93.1% of undergrads are < 24 years old 94.6% of undergrads are full-time students

5. Illinois State University: Who We Are Fact Book (Planning and Institutional Research) – Fall 2010 First-year students ACT mean composite scores: – Illinois State 24.1 – State of Illinois20.7 – National21.0

6. Illinois State University: Who We Are Transfer Students – 1,838 new transfer students in Fall 2010 – ~50% of bachelor’s degrees granted annually 85.0% retention from first-year to sophomore

7. University Assessment Services Illinois State University UAS Office of the Provost IRFinance and Planning The two offices work closely; however, UAS is responsible for the management of all program-level assessment.

8. History of ISU’s Gen Ed Program Senate approval in 1992 Full implementation in 1998 Assessment task force in 2005 Full implementation in 2008 New task force in 2011-2012 – Assessment subcommittee!

9. Current Gen Ed Program 12 Goals with 40 distinct skills/abilities 42 credit hours 190 potential courses Approximately 13,410 enrollments per semester 3 Cores: – Inner – (5 courses) – Middle – (5 courses) – Outer – (4 courses)

10. Current Gen Ed Assessment Comprehensive, yet manageable Purpose: To provide the Council for General Education with sound evidence to base decisions regarding the program.

11. Important Contributors AKA Campus Buy-In Director of General Education Council for General Education General Education Assessment Task Force Assessment Advisory Council

12. Research Reviewed past methodologies employed at ISU Reviewed methodologies employed at other institutions

13. Initial Challenge Goals of General Education – Numerous (12 with 40 abilities) – Difficult for faculty/staff/students to remember and reference – In some cases, challenging to measure

14. Solution Four Shared Learning Outcomes Common and integrated elements of the established goals of Gen Ed Also aimed at eliminating some of the division that is present between Gen Ed and the Major

16. Selecting an Assessment Method: The Institutional Portfolio (Seybert) Principles – Gen Ed is responsibility of entire campus – Be minimally intrusive (both faculty and students) – Use existing student work

17. Selecting an Assessment Method: The Institutional Portfolio (Seybert) Process – “Artifacts” are collected – Faculty teams review using rubrics – Results are compiled and reported to committee – Committee makes decisions based on results

18. Selecting an Assessment Method: The Institutional Portfolio (Seybert) Characteristics – Faculty review teams are multidisciplinary – Review is invisible to students and not intrusive to faculty – Process is labor intensive and requires resources – Process is dynamic and “messy”

19. Method – Requirements

20. Non-intrusive Cover the 12 goals of general education Comprehensive, Manageable Institution-focused

21. IAP Schedule Fall 2008Public Opportunity Spring 2009Critical Inquiry and Problem Solving Fall 2009Diverse and Global Perspectives Spring 2010Life-Long Learning Fall 2010Critical Inquiry and Problem Solving Spring 2011Public Opportunity Fall 2011Life-Long Learning Spring 2012Diverse and Global Perspectives

22. General Overview for Assessment Phase One: Obtaining the artifacts Phase Two: Sampling the artifacts Phase Three: Reviewing the artifacts

23. Phase One: Obtaining the Artifacts Instructor Participation: 1.Read the invitation letter from Provost 2.Identify artifact(s) that address at least 1 Primary Trait 3.Complete the online participation form 4.Allow UAS access to the artifacts for up to 24 hours

24. Shared Learning Outcome – Public Opportunity (Students will identify the resources and articulate the subsequent value of civic and community engagement.) Primary Traits for Public Opportunity: 1.Critically informed position on civic life 2.Influence of civic participation on the social and collaborative nature of knowledge 3.Contributions to the public affecting individual life aspects- such as family, religion, business and/or the state 4.Contributions to the public affecting social and community life aspects – such as family, religion, business and/or the state 5.Resources for civic engagement 6.Civic participation in the social, economic, technological, and/or political dimensions of community development 7.Self-Reflection (Elective) 8.Discipline Knowledge (Elective)

25. Phase Two: Sampling the Artifacts Goal – Obtain a random sample of 100 artifacts per core (300 total artifacts per Shared Learning Outcome) Behind the Scenes at UAS: Calculate the total number of enrollments, based on class size Remove all identifying information and oversample End of the term, recalculate and use the appropriate proportions

26. Sampling Example Course Dept. & NumberInstructorCoreSection # # of artifacts # of students% of total # needed for final sample # to sample and copy ABC 200AOuter21638.11816 BCD 100BOuter2116220.852142 BCD 100COuter12496.31612 BCD 100COuter3111715.061530 DEF 200DOuter61293.7348 BCD 200EOuter41455.79612 BCD 200FOuter11435.53510 EFG 100GOuter13455.7964 EFG 200HOuter11232.9636 EFG 200GOuter21303.8648 EFG 200GOuter31212.7036 DEF 200IOuter31293.7348 DEF 200JOuter71273.4736 FGH 200KOuter11455.79612 BCD 100LOuter41496.31612 TOTAL 777100.00100 200

27. Sampling Example Course Dept. & NumberInstructorCoreSection # # of artifacts # of students% of total # needed for final sample # to sample and copy ABC 200AOuter21638.11816 BCD 100BOuter2116220.852142 BCD 100COuter12496.31612 BCD 100COuter3111715.061530 DEF 200DOuter61293.7348 BCD 200EOuter41455.79612 BCD 200FOuter11435.53510 EFG 100GOuter13455.7964 EFG 200HOuter11232.9636 EFG 200GOuter21303.8648 EFG 200GOuter31212.7036 DEF 200IOuter31293.7348 DEF 200JOuter71273.4736 FGH 200KOuter11455.79612 BCD 100LOuter41496.31612 TOTAL 777100.00100 200

28. Sampling Example Course Dept. & NumberInstructorCoreSection # # of artifacts # of students% of total # needed for final sample # to sample and copy ABC 200AOuter21638.11816 BCD 100BOuter2116220.852142 BCD 100COuter12496.31612 BCD 100COuter3111715.061530 DEF 200DOuter61293.7348 BCD 200EOuter41455.79612 BCD 200FOuter11435.53510 EFG 100GOuter13455.7964 EFG 200HOuter11232.9636 EFG 200GOuter21303.8648 EFG 200GOuter31212.7036 DEF 200IOuter31293.7348 DEF 200JOuter71273.4736 FGH 200KOuter11455.79612 BCD 100LOuter41496.31612 TOTAL 777100.00100 200

29. Phase Three: Reviewing the Artifacts 1.Instructors solicited to apply to be IAP reviewers 2.Interdisciplinary two-person teams (3 per Shared Learning Outcome and an alternate) are trained in use of established rubrics 3.Review teams complete consensus analysis using developed rubrics for the 4 Shared Learning Outcomes

30. Review Process Review Week: Day One: – Reviewers interpret the rubrics as a group – Calibration training using practice artifacts Days Two through Four: – Reviewers split into teams and complete their binders – Final rubrics collected throughout

31. Phase Three: Reviewing the Artifacts

33. Excerpt from the Public Opportunity Rubric: Primary Traits Not PresentDevelopingEstablishedAdvanced Gen Ed Goal Critically informed position on civic life Describes the value of contributions to civic life in the dimensions of their own life Compares the value of contributions to civic life from multiple, critically informed perspectives Defends or refutes the value of contributions to civic life 7.c Influence of civic participation on the social and collaborative nature of knowledge Identifies how civic participation can change the social and collaborative nature of knowledge Explains how civic participation can change the social and collaborative nature of knowledge Applies new knowledge in the context of civic participation 12

34. Phase Three: Reviewing the Artifacts 1.Instructors solicited to apply to be IAP reviewers 2.Interdisciplinary two-person teams (3 per Shared Learning Outcome and an alternate) are trained in use of established rubrics 3.Review teams complete consensus analysis using developed rubrics for the 4 Shared Learning Outcomes 4.The final data are reported to Council of General Education to formulate commendations/recommendations

35. Results

36. Reliability – Inter-rater reliability (calibration artifacts) Percent agreement – Public Opportunity: 42% - 71% – Critical Inquiry and Problem Solving: 61% - 69% Intraclass correlation coefficient – Diverse and Global Perspectives:.70 -.91 – Life-Long Learning:.26 -.63 – Test-retest reliability (repeat calibration artifacts) Exploratory factor analysis

37. Results Focus on trends for each Shared Learning Outcome No consistent differences across freshmen, sophomores, juniors, and seniors No clear differences emerged when comparing native and transfer students

38. Results Maximum review value consistent with instructions

39. Discussion Limitations of the results – Work in progress – Expand from 3 Cores to many course categories – Data based on courses from one semester only

40. Discussion Future directions – More time calibrating during review process – Reports with data from two different semesters – General Education Task Force (and its Assessment subcommittee)

41. Conclusions Advantages of the IAP – Nonintrusive to instructional faculty and staff and students – Allows for program-level assessment – Maintains the institution as the focus

42. Conclusions Challenges of the IAP – Faculty participation – Locus of generalizability – Takes time

43. Conclusions Lessons learned – Patience and public relations – Streamline the process – Oversample – Reviewer calibration is key – Alternate reviewer – Closing the loop

44. QUESTIONS?

45. For More Information http://gened.illinoisstate.edu/ http://assessment.illinoisstate.edu/generaleducation/ http://assessment.illinoisstate.edu/about/newsletter.shtml

46. Illinois State University University Assessment Services Normal, IL 309.438.2135 assessment@ilstu.edu Renée M. Tobin, Ph.D. Derek J. HerrmannKelly L. Whalen Acting Director Coordinator Graduate Assistant rmtobin@ilstu.edurmtobin@ilstu.edu djherrm@ilstu.edu klwhal2@ilstu.edudjherrm@ilstu.eduklwhal2@ilstu.edu