Whats Going On? Five Year Reviews and Assessment: Sharing Experiences so Far Ruth Guzley, Mike Ward and Bill Loker, AURA

Five Year Program Reviews: The Broader Context … Were routine, episodic efforts of little utility, much futility What do we want to know about our programs? What are we trying to accomplish? How well are we meeting our goals? Academic success, good use of resources

Assessment: Doing the Right Thing for the Right Reasons The Intentional Curriculum: Do individual courses add up to a program? The Reflective Department: Are we working together to provide a quality educational experience for our students? The Culture of Evidence: How do we know the answers to the above questions? Assessment can provide these answers …

The Assessment Juggernaut WASC reaccredidation … educational effectiveness reviews … Cornerstones … CSU system-wide accountability effort focused on quality of Baccalaureate programs … Educational effectiveness and other stakeholders … students, parents, legislators, etc

Aligning Five Year Review with Assessment Demands … New Five Year Review reflects assessment concerns Transforms FYR from an episodic time sink into a cumulative exercise in evidence gathering to improve educational outcomes and satisfy stakeholder accountability demands Assessment only one component of FYR

From the FYR template … Articulating a Collective Vision: Mission and Program goals Criteria for Review 1.1 The program has a clear and published mission statement that is appropriate for higher education and consonant with the mission and strategic priorities of the university. 1.3 The program periodically reviews and revises the mission statement as appropriate. The review process involves appropriate stakeholders. Program Performance (Hint: Insert mission statement here.)

From the FYR template … (contd) Organizing For Learning: Curriculum Criteria for Review 2.2 The program has specified its expected learning outcomes and they have been widely shared among its members, including faculty, students, staff, and … external stakeholders. 2.5 The program has established processes for assessing student learning outcomes and for assuring that students are achieving core competencies for completion of the program. Program Performance … (Hint: you dont want a blank here.)

From the FYR template … (contd) Organizing For Learning: Faculty Resources Criteria for Review 3.1 Faculty recruitment, incentives and evaluation practices are aligned with … program missions and educational objectives. 3.3 The deployment of faculty reflects the mission and program goals … Program Performance … (Hint: Need mission, goals and outcomes statements to answer these questions.)

The FYR is consequential 3. Final Review and Approval of Continuous Approval Plan The Review Team – consisting of an Office of the Provost representative, College Dean, Department Chair, Program Director, and an undergraduate student – will make one of four recommendations to the Provost …. 1.Recommend continuation … 2.Recommend continuation, but cite specific concerns … 3.Recommend the program remain under continuing review …. 4.Recommend suspension …. (Hint: Outcome #4 not so good …)

The Right Thing, Right Reasons, & Get a Jump on FYR Aligning Mission, Goals, Student Learning Outcomes and Curriculum is Good Practice Best Practice engages in a process of continuous assessment & improvement SLO assessment is a necessary component of FYR … you dont want blanks! Departments do things other than teach, but excellence in undergraduate education is our #1 strategic priority … Assessment of SLOs: evidence of success

Toward A Comprehensive Assessment Plan: Program Mission Goals Student Learning Outcomes (SLOs) Course Alignment Matrices

An Outcomes / Course Alignment Matrix CourseSLO 1SLO 2SLO 3SLO 4 151II 177PI 201PI 277PP 340M - oral presentation M 477P - term paper Stage 1 – Ensuring a Cohesive Curriculum In which courses are particular outcomes introduced (I), practiced (P), mastered (M)?

Assessment of SLOs Direct Assessment Methods: Published Tests Local Exams Embedded Assignments & Activities Portfolios Indirect Assessment Methods: Surveys Interviews Focus Groups

Assessment of SLOs Direct Assessment Methods: Examine Student Work … Indirect Assessment Methods: Examine Student / Alumni Perceptions … Effective Assessment Programs Use Multiple Measures … Triangulate

Using Rubrics to Objectively Evaluate Student Work … Explicit Scoring for Student Work: Holistic Scoring Analytical Rubrics Program Faculty Should Discuss, Debate, Develop Some Consensus on What Methods and Measures (Rubrics) to Use …

Evaluation and Use of Data for Continuous Improvement Assessment Data Must be Evaluated Systematically, Periodically Recommend Annual Collection / Assessment / Action Plan Modify Curriculum and Pedagogy to Improve Student Learning … Typical Actions : Modify Content, Sequencing of Course(s) Modify Pedagogical Approaches Follow-up Focus Group or Survey to Triangulate …

Moving Toward Continuous Improvement: The Entire Process (Goals, Outcomes, Assessment Measures) Must be Evaluated Periodically Recommend Five Year Re-evaluation to synchronize with FYR process Are we measuring reasonable work samples? Is our process meaningful? Are we better informed about what our students are learning or not? Is our process sustainable? Can we be more efficient and still have valid measures?

The Assessment Plan Design Process … Wherever you begin developing your Assessment Program will NOT be the ideal place to start … You Must Begin Somewhere … Iteration will save you... Continuous Refinement of Methods Consensus and Team Building Alignment of Program Resources with Program Goals and Faculty Values …

Case Study: Mechanical Engineering Program Mission Mechanical Engineering Program Mission Statement The Mechanical Engineering Program has the primary mission of providing a high-quality undergraduate engineering education with 1.A curriculum that is firmly grounded in engineering fundamentals 2.A faculty that provides superior teaching and mentoring both in and out of the classroom 3.A faculty whose focus is undergraduate education 4.Class sizes that encourage student participation 5.Project experiences that build on fundamentals and develop team skills 6.Facilities and equipment that are readily accessible 7.An environment that is conducive to learning and encourages students from different genders and backgrounds Note: Not Goals or Outcomes for Student Learning !

Mechanical Engineering Program graduates demonstrate the ability to: A.Apply knowledge of math, science, and mechanical engineering to formulate and solve problems. B.Conduct experiments, and analyze and interpret the resultant data. C.Design mechanical components and systems to meet desired needs, and design experiments to evaluate system performance with respect to specifications. D.Use current mechanical engineering tools. E.Communicate technical matters effectively in oral, written, and graphical form. F.Function effectively as members of multi-disciplinary teams. G.Understand their professional and ethical responsibilities, the impact of their activities on society and the environment, and appreciate contemporary issues facing society. H.Use information resources and recognize the importance of continued learning. Mechanical Engineering Student Learning Outcomes

Mechanical Engineering Embedded Assessment Matrix: SLOs: A B C D E F G H Courses where student work is measured

Embedded Assessment of Outcome C in ME 138: Design mechanical components and systems to meet desired needs … Student grades on design projects are based on assessment of 6 design projects: 3 individual, 3 group A standardized rubric is used to score each design The performance metric: A cumulative grade of C or better on design projects must be attained to demonstrate the ability to design mechanical components and systems (competence).

Other Measures of Mechanical Engineering Outcome C (Triangulation!) Senior Exit Survey Question: How well prepared are you to design a component or system to meet desired needs? Mean Responses (4.29 Spring 2002, 4.0 Spring 2003) Scale: 5=Very Well Prepared – 1=Very Unprepared A mean response of 4.0 is considered acceptable to faculty Outcome C: Design mechanical components and systems to meet desired needs …