ODU Faculty Development March 18, 2011 Academic Assessment ODU Faculty Development March 18, 2011

Plan for the day Update on changes in Higher Learning Commission Update on Academic Assessment Webpage Program Assessment Survey Results 2009-2010 & 2010-2011

Plan for the day Best Practices in Program Outcomes Course Matrix Development & Utilization Open Forum: Posters from 2009-2010 Focus Groups 2010-2011 Focus Groups Meet

Higher Learning Commission Update Linda Schoen

Update on Academic Assessment Committee Work Anna Parkman

What we’ve been doing Program Assessment Survey Analysis & Comparisons 2009-2010 2010-2011 Designing Webpage for Academic Assessment Completing and submitting HLC AA Report for Jan 2011 Submission Working with 2010-2011 Focus Groups

Program Assessment Survey Results 2009-2010 & 2010-2011 What we’ve learned Next Steps Questions

Please evaluate the current state of your program mission statement. Answer Options Response 2010 Response 2009 We do not have a program level mission statement at this time. 0.0% Our program mission statement has been developed, but it needs work. 48.0% 51% Our program mission statement has been developed and we are satisfied with it. 52.0% 49 2010 results updated fall 2011 to include late submissions.

Please evaluate the current state of your student learning outcomes. Answer Options Response 2010 Response 2009 We don't have student learning outcomes developed at this time. 0.0% 6% Our student learning outcomes are developed, but need significant work. 4% Our student learning outcomes are developed, but need review by faculty & some minor revisions. 72.0% 66% Our student learning outcomes are developed and are in good shape. 24.0% 22% 2010 results updated fall 2011 to include late submissions

We don't have an outcome course matrix developed at this time. 16% 24% Please evaluate the current state of your program outcome course matrix. Answer Options Response 2010 Response 2009 We don't have an outcome course matrix developed at this time. 16% 24% An outcome course matrix has been developed, but needs significant work. 12% The outcome course matrix has been developed, but needs a review by the program faculty and perhaps some minor revisions. 58% 50% The outcome course matrix is well developed and is in good shape. 13% 15% 2010 results updated fall 2011 to include late submissions

Please evaluate the current state of your student learning outcome measures. Answer Options Response 2010 Response 2009 We don't have student learning outcome measures developed at this time. 24% 29% Student learning outcome measures have been developed, but never utilized to collect data. 16% 14% Student learning outcome measures have been employed and data collected, but never utilized. 12% 9% Student learning outcome measures have been employed, data has been collected and has been utilized. 48% 49 2010 results updated fall 2011 to include late submissions

Program Learning Outcomes Best Practices

What are program learning outcomes? Formal statements that articulate: What students are able to at the completion of the program Program Goals vs. Outcomes Process/Fluid

Best Practices Simply put Know what you are doing Know why you are doing it Know what students are learning as a result Changing because of that information

Writing Learning Outcomes Learning Outcomes Formula Bloom’s Taxonomy Characteristics of Good Learning Outcomes

1. What do you want the student to be able to do? This question asks you to develop the outcome. For Example: Student identifies, consults and evaluates reference books appropriate to the topic in order to locate background information and statistics. ACRL’s IIL Immersion Summer 2005

Learning Outcomes Formula Verb Or Action Phrase Great Learning Outcomes + Why? = “In Order To” OR What students need to know? “Student identifies, consults and evaluates reference books appropriate to the topic” Why do they need to know this? “locate background information and statistics.” “In Order To” ACRL’s IIL Immersion Summer 2005

Bloom’s Taxonomy

Characteristics of an Effective Outcome A good outcome statement Consists of action verbs Uses simple language Describes student rather than instructor behaviors Focuses on the end result

Characteristics of an Effective Outcome Is measurable Is limited to a single outcome Is attainable by the end of the program Is clear to those outside the discipline Can be validated by colleagues

Problematic Program Outcomes The graduate develops the knowledge, experience and abilities required to be successful in this fascinating world of art history. identifies competing values in business decisions and can act with a commitment to integrity in interactions that lead to the development of trustworthy relationships with stakeholders - customers, colleagues, suppliers and other key business partners.

Example 1: Strong Set of Outcomes Accurately interprets the biological basis of phenomena while recognizing the limitations of scientific analysis Demonstrates proficiency in data gathering. Designs, conducts and communicates biological experiments Applies concepts of biology to societal issues

Review and Discussion of Your Program Outcomes at Your Table

Review your program outcomes Are there “and” statements that make the statement too confusing? Are these higher order bloom statements? Can you get them down to 5-8 outcome statements? Can you measure them? What will you do next?

A Great Resource http://www2.acs.ncsu.edu/UPA/assmt/resource.htm

The Program Learning Outcomes/Course Matrix Jim Cottrill

Purpose of the outcomes/course matrix Planning: Are the components necessary for student success in place and correctly ordered? Assessment: Has everything that we ought to measure been measured? Is there a coordinated effort to assess SLO across courses from introduction to mastery?

Two Types of Matrices for Planning and Assessment Course level matrix maps KSDs to particular assessment items planned for a course. Used to ensure that all KSDs are assessed Visually depicts the “balance” of the course

Two Types of Matrices for Planning and Assessment (cont.) Program level matrix maps SLOs to the courses required in the program. Assumes a mapping between course KSDs and program SLOs Maps progress from introduction to reinforcing to mastery of outcome (I recommend using I, R, M) May identify lack of balance in levels, such as mastery in 100 level course or introduction in Capstone

Some examples:

Some examples:

S2 To develop students’ expertise in specific technology usage. CIS 150 CIS 180 CIS 234 CIS 316 MTH 110 MTH 145 PHL 101 Knowledge 1.1 X 1.2 1.3 2.1 2.2 2.3 3.1 3.2 3.3 3.4 4.1 4.2 4.3 S2 To develop students’ expertise in specific technology usage. 2.1 Students will develop a proficiency with database management systems. 2.2Students will be able to explain various network topologies and protocols. 2.3Students will be able to analyze network graphs to determine efficient paths.

Outcome Matrices Division of Business Overall I=Introduce R=Reinforce E=Emphasize HST 106 107 201 202 231 232 330 498 Students will develop the ability to read and write critically about historical subjects R R/E Students will develop the ability to research, to organize material, and to present their findings clearly and cogently Students will develop the ability to analyze and synthesize various approaches to history, and primary and secondary documents I/R

Review & Discussion of Your Matrix at Your Table Do You have one? What do you need to update/change? Next Steps

What do program faculty do in the “focus group” year Math Biology Env With Time & Energy What do program faculty do in the “focus group” year Math Biology Env

FAQ What do program faculty do in the “focus group” year Math ppt Biology ppt

Focus Groups Work One Program Mission Statement Program Outcomes Statements Program Outcomes/Course Matrix Outcome Measures Utilization 5-Year Assessment Plan

Closing 2011-2012 PAS Program Outcomes Program /Outcomes Matrix Focus on measurement tools