Institutional Assessment: Connecting the Puzzle Pieces MSCHE Presentation December 11, 2011 Connecting the Puzzle Pieces MSCHE Presentation December 11, 2011 Christine Licata Senior Associate Provost RIT Anne Wahl Director of Student Learning Outcomes Assessment RIT

Workshop Outcomes 1. Self-assess infrastructure design and processes 2. Review program level assessment framework 3. Determine which tools and strategies to adopt or adapt 1. Self-assess infrastructure design and processes 2. Review program level assessment framework 3. Determine which tools and strategies to adopt or adapt

Institutional Assessment Major “pieces” of the puzzle: 1. Corner pieces Infrastructure 2. Center pieces: Student learning outcomes assessment Administrative unit effectiveness Major “pieces” of the puzzle: 1. Corner pieces Infrastructure 2. Center pieces: Student learning outcomes assessment Administrative unit effectiveness

Institutional Assessment Infrastructure - 4 Corner Pieces

Key Results Area #3: Demonstrate Educational Excellence 11a. Meet/exceed outcome goals for program-level student learning Performance Commitment FY 2009FY 2010FY 2011FY 2012FY 2013 ActGoalActGoalActGoalActGoalActGoal i. Expected program student learning achievement levels met or exceeded. NA 40%NA 55%NA65%NA75% ii. Assessment results and processes guide planning and improvement. NA 90%85%NA85%NA85%NA100% Corner Piece: Strategic Vision RIT’s Strategic Plan Corner Piece: Strategic Vision RIT’s Strategic Plan

Output Indicators: Learning Outcomes Corner Piece: Strategic Vision - Provost’s Academic Quality Dashboard Corner Piece: Strategic Vision - Provost’s Academic Quality Dashboard

Three Central Committees 1. Student Learning Outcomes Assessment Committee (SLOAC) (Academic Programs) 2. University Assessment Council (UAC) (Administrative Units) 3. General Education Committee (GEC) (General Education Framework) Three Central Committees 1. Student Learning Outcomes Assessment Committee (SLOAC) (Academic Programs) 2. University Assessment Council (UAC) (Administrative Units) 3. General Education Committee (GEC) (General Education Framework) Corner Piece: Governance Structure Corner Piece: Governance Structure

1. Student Learning Outcomes Assessment Committee (SLOAC) Nine colleges represented Meet monthly Formal charge & work plan Key Support Areas:  Program level assessment  Annual progress report 1. Student Learning Outcomes Assessment Committee (SLOAC) Nine colleges represented Meet monthly Formal charge & work plan Key Support Areas:  Program level assessment  Annual progress report Corner Piece: Governance Structure Corner Piece: Governance Structure

2. University Assessment Council (UAC) All divisions represented Meet every-other month Formal charge & needs assessment Key support areas:  Institutional Effectiveness Map  Annual Progress Report 2. University Assessment Council (UAC) All divisions represented Meet every-other month Formal charge & needs assessment Key support areas:  Institutional Effectiveness Map  Annual Progress Report Corner Piece: Governance Structure Corner Piece: Governance Structure

3. General Education Committee (Standing Committee of Academic Senate) Nine colleges represented Meet weekly Formal charge Key support areas:  General Education Framework  General Education Assessment Plan 3. General Education Committee (Standing Committee of Academic Senate) Nine colleges represented Meet weekly Formal charge Key support areas:  General Education Framework  General Education Assessment Plan Corner Piece: Governance Structure Corner Piece: Governance Structure

Centralized Support and Leadership for Assessment One office responsible for Student Learning Outcomes Assessment – in Provost’s Office Permanent budget Building capacity - staff: Director, Senior Assessment Associate, Assessment Management System Coordinator, and Senior Staff Assistant Director co-chairs UAC and is a member of SLOAC, GEC, Data Warehouse Steering Committee, Survey Council Developing resources and website Faculty stipends Corner Piece: Leadership and Coordination Corner Piece: Leadership and Coordination

Corner Piece: Leadership and Coordination Corner Piece: Leadership and Coordination

Examples of Existing Resources Institutional Research and Policy Studies Office Data warehouse and student surveys Teaching and Learning Center Workshops and events for faculty Cooperative Education Office Co-op evaluation Assessment Management System (TaskStream) Corner Piece: Existing Resources and Data Sources

Assessment ResourcesMiddle States Standards ASQ Admitted Student Questionnaire8 Program-level Assessment of Student Learning14 Co-operative Experience Evaluation14 Delaware Study10 Campus Climate Survey7,9 Employee Satisfaction Survey7 CSI – Freshmen Survey7, 9 NSSE1, 7, 9, 12 Noel Levitz Student Satisfaction Survey1, 7, 9 Academic Program Review (2015)1,2,11 Alumni Survey7, 11 Student-to-Student Survey9 Enrollment Budget Projections and Forecasting2, 3 Internal Audits3 Space Utilization Study3 Student Evaluations of Teaching7, 10 Institutional Research Data1, 2, 7 Samples of Existing Data Sources Corner Piece: Existing Resources and Data Sources

Center Piece: Program-Level Assessment Framework Program-Level Student Learning Outcomes: Planning Implementing Using Results Sustaining Program-Level Student Learning Outcomes: Planning Implementing Using Results Sustaining

Center Piece: Program-Level Student Learning Outcomes Planning University-wide engagement: Student Learning Outcomes Assessment Committee (SLOAC) Common language: Program-Level Assessment Plan Planning University-wide engagement: Student Learning Outcomes Assessment Committee (SLOAC) Common language: Program-Level Assessment Plan

Program Goals Student Learning Outcomes Academic Program Profile Data Source Measure Curriculum Mapping BenchmarkTimeline Data Analysis Key Findings Use of Results Action Items and Dissemination List program- level goals Students will be able to: (task, capability, knowledge, skills, and dispositions) Use measurable verbs. Align to the RIT essential outcomes Check all that apply.  Critical Thinking  Ethical Reasoning  Integrative Literacies  Global Interconnectedness  Creative/Innovative Thinking Assessment opportunity Course or Experience Method measures Assignment rubric Standard, target, or achievement level (usually a %) Statement of student success Identify when and how data are collected aggregated and analyzed Identify who is responsible and list key findings Identify how results are used and shared. List recommendations or action items Center Piece: Program-Level Assessment Plan

Implementing Building relationships: from 1:1 to “moving in” Leveraging accreditation Providing support and resources for :  Benchmarking  Designing instruments  Developing rubrics  Conducting data analyses Implementing Building relationships: from 1:1 to “moving in” Leveraging accreditation Providing support and resources for :  Benchmarking  Designing instruments  Developing rubrics  Conducting data analyses Center Piece: Program-Level Student Learning Outcomes

Use of Results RIT Annual Progress Report (3 samples) College-level action plans SLOA office resources/support Use of Results RIT Annual Progress Report (3 samples) College-level action plans SLOA office resources/support Center Piece: Program-Level Student Learning Outcomes

Progress Report Sample Emerging Assessment Program BS Criminal Justice P ROGRAM OUTCOME 1P ROGRAM OUTCOME 2 Program Level Student Learning Outcome (SLO) Demonstrate knowledge of key concepts, policy issues, legal and ethical issues surrounding crime and justice Utilize critical thinking skills to apply knowledge to social issues in crime and criminal justice policies Assessment Method Course: Seminar in Criminal Justice Final Examination (mapped to SLO) Field Experience Capstone Paper (mapped to SLO) Achievement Benchmarks and Results Benchmark: 70% of students will earn a grade of C or better Results: 81% met goal Benchmark: 70% will earn a grade of C or better Results: 100% met goal Use of Results  Faculty recommend moving from 1 comprehensive final to mini-exams to provide additional assessment opportunities and feedback.  Re-assessment in spring  Faculty to review rubric and rating process to determine alignment of rubric to SLO and consistency of grading

Progress Report Sample Developed Assessment Program BS Game Design and Development P ROGRAM OUTCOME 1P ROGRAM OUTCOME 2 Program Level Student Learning Outcome(SLO) Apply current technical concepts and practices within the computing disciplines to the field of game design and development Incorporate accepted game industry processes towards the construction of a game design and development application or prototype Assessment Method -Programming for Digital Media Course- selected Homework and Project rubric (mapped to both SLOs) Achievement Benchmarks and Results Benchmark: 80% will achieve a 3 (acceptable) or higher average score across rubric items Results: 83% met goal Benchmark: 80% will achieve a 3 (acceptable) or higher average score across rubric items Results: 63% achieved 3 Use of Results  Data was disseminated to faculty & Chair.  Results discussed at faculty retreat.  No action required as benchmark was met, but information was utilized in the Semester Conversion Process.  Analysis showed the content was too condensed at week 8- limited time for polished prototype experience. Course modified- content spread over 10 weeks.  Re-assessment will take place in next cycle.

Progress Report Sample Highly Developed Assessment Program BS Mechanical Engineering P ROGRAM OUTCOME 1P ROGRAM OUTCOME 2 Program Level Student Learning Outcome (SLO) Work independently and collaboratively while demonstrating the professional and ethical responsibilities of the engineering profession. Enhance skills through formal education and training, independent inquiry, and professional development. Assessment Method  Employer Evaluation of Co-op Experience  Student Co-op Self-Evaluation Student Co-op Self-Evaluation: students rated need/value of continuous learning Achievement Benchmarks and Results Benchmark: 100% will earn a 4 or above Results: 100% scored 4 or above by employers in: ethics, teamwork, leadership, respect. Students scored themselves above 3, but below the levels reported by employers (4). Benchmark: 100% rank this as a 4 or above in terms of importance Results: 100% of students ranked the importance level as a 4 or higher Use of Results  Discrepancy between student and employers suggest a need to offer more opportunities to develop team building and leadership skills.  Further discussion by faculty in fall 2011 to determine how further skill building in teamwork and leadership can be intentionally incorporated into curriculum. Results suggest students understand and feel prepared for continuous learning throughout career. Results are guiding strategic decisions at the department level about the future of engineering education and approach to marketplace needs.

Sustaining Technology  Assessment Management System  Website Integrated processes  Academic program review  Annual reporting Communication - r egular sharing and dissemination of findings and use of results Closing the Loop  Input to budget  Informs strategic planning Sustaining Technology  Assessment Management System  Website Integrated processes  Academic program review  Annual reporting Communication - r egular sharing and dissemination of findings and use of results Closing the Loop  Input to budget  Informs strategic planning Center Piece: Program-Level Student Learning Outcomes

Center Piece: Administrative Unit Effectiveness University Assessment Council (UAC) Assessment Management System Institutional Effectiveness (IE) Map Next Steps:  AMS Annual Report  Develop unit audit University Assessment Council (UAC) Assessment Management System Institutional Effectiveness (IE) Map Next Steps:  AMS Annual Report  Develop unit audit

Administrative Unit:Contact Name:Academic Year: Mission: Administrative Unit Objectives or Outcomes Division Goal (If Applicable) Strategic Plan Key Results Area Align to the RIT’s Key Results Areas - check all that apply OR list other Strategic Initiative. Data Source Unit Activities Method/ Measure Describe metrics or instruments used to measure results Benchmark or Standard Statement of Success/ Measureable Outcome Timeline & Person(s) Responsible Indicated Year, Cycle and/or Semester Achievement of Outcome/ Objective Indicate level of achievement Next Steps or Actions What, if any, steps or actions will take place as a result of the assessment findings  KRA 1 Renowned for Student Success  KRA 2 Maximize Opportunities for Innovation, Creativity, Research and Scholarship  KRA 3 Execute with Organizational and/or Operational Excellence  KRA 4 Achieve Highest Levels of Stakeholder Satisfaction  Other Strategic Initiative - please list  Not Met  Approaching  Met  Exceeded Center Piece: Administrative Unit Effectiveness: IE Map

Self-assessment tool for institutions Center Piece: Institutional Assessment Self-Assessment

Questions